Hyper Temporal Lidar with Multi-Channel Readout of Returns

ABSTRACT

A lidar receiver can employ multiple readout channels that are capable of simultaneously reading out sensed signals from different pixel sets of a photodetector array in order to detect different returns from different laser pulse shots. In doing so, the lidar receiver can support the use of overlapping detection intervals when collecting signal data for detecting the different returns from the different laser pulse shots.

CROSS-REFERENCE AND PRIORITY CLAIM TO RELATED PATENT APPLICATIONS

This patent application claims priority to U.S. provisional patentapplication 63/186,661, filed May 10, 2021, and entitled “Hyper TemporalLidar with Controllable Detection Intervals”, the entire disclosure ofwhich is incorporated herein by reference.

This patent application also claims priority to U.S. provisional patentapplication 63/166,475, filed Mar. 26, 2021, and entitled “HyperTemporal Lidar with Dynamic Laser Control”, the entire disclosure ofwhich is incorporated herein by reference.

This patent application is related to (1) U.S. patent application Ser.No. ______, filed this same day, and entitled “Hyper Temporal Lidar withControllable Detection Intervals” (said patent application beingidentified by Thompson Coburn Attorney Docket Number 56976-213637), (2)U.S. patent application Ser. No. ______, filed this same day, andentitled “Hyper Temporal Lidar with Shot-Specific Detection Control”(said patent application being identified by Thompson Coburn AttorneyDocket Number 56976-213638), (3) U.S. patent application Ser. No.______, filed this same day, and entitled “Hyper Temporal Lidar withControllable Detection Intervals Based on Range Estimates” (said patentapplication being identified by Thompson Coburn Attorney Docket Number56976-213639), (4) U.S. patent application Ser. No. ______, filed thissame day, and entitled

“Hyper Temporal Lidar with Controllable Detection Intervals Based onEnvironmental Conditions” (said patent application being identified byThompson Coburn Attorney Docket Number 56976-213640), (5) U.S. patentapplication Ser. No. ______, filed this same day, and entitled “HyperTemporal Lidar with Controllable Detection Intervals Based on Regions ofInterest” (said patent application being identified by Thompson CoburnAttorney Docket Number 56976-213641), (6) U.S. patent application Ser.No. ______, filed this same day, and entitled “Hyper Temporal Lidar withControllable Detection Intervals Based on Location Information” (saidpatent application being identified by Thompson Coburn Attorney DocketNumber 56976-213642), (7) U.S. patent application Ser. No. ______, filedthis same day, and entitled “Hyper Temporal Lidar with OptimizedRange-Based Detection Intervals” (said patent application beingidentified by Thompson Coburn Attorney Docket Number 56976-213643), (8)U.S. patent application Ser. No. ______, filed this same day, andentitled “Hyper Temporal Lidar with Multi-Processor Return Detection”(said patent application being identified by Thompson Coburn AttorneyDocket Number 56976-213644), (9) U.S. patent application Ser. No.______, filed this same day, and entitled “Bistatic Lidar Architecturefor Vehicle Deployments” (said patent application being identified byThompson Coburn Attorney Docket Number 56976-213646), and (10) U.S.patent application Ser. No. ______, filed this same day, and entitled“Hyper Temporal Lidar with Asynchronous Shot Intervals and DetectionIntervals” (said patent application being identified by Thompson CoburnAttorney Docket Number 56976-213647), the entire disclosures of each ofwhich are incorporated herein by reference

INTRODUCTION

There is a need in the art for lidar systems that operate with lowlatency and rapid adaptation to environmental changes. This isparticularly the case for automotive applications of lidar as well asother applications where the lidar system may be moving at a high rateof speed or where there is otherwise a need for decision-making in shorttime intervals. For example, when an object of interest is detected inthe field of view for a lidar transmitter, it is desirable for the lidartransmitter to rapidly respond to this detection by firing highdensities of laser pulses at the detected object. However, as the firingrate for the lidar transmitter increases, this places pressure on theoperational capabilities of the laser source employed by the lidartransmitter because the laser source will need re-charging time.

This issue becomes particularly acute in situations where the lidartransmitter has a variable firing rate. With a variable firing rate, thelaser source's operational capabilities are not only impacted by periodsof high density firing but also periods of low density firing. As chargebuilds up in the laser source during a period where the laser source isnot fired, a need arises to ensure that the laser source does notoverheat or otherwise exceed its maximum energy limits.

The lidar transmitter may employ a laser source that uses opticalamplification to support the generation of laser pulses. Such lasersources have energy characteristics that are heavily impacted by timeand the firing rate of the laser source. These energy characteristics ofa laser source that uses optical amplification have importantoperational impacts on the lidar transmitter when the lidar transmitteris designed to operate with fast scan times and laser pulses that aretargeted on specific range points in the field of view.

As a technical solution to these problems in the art, the inventorsdisclose that a laser energy model can be used to model the availableenergy in the laser source over time. The timing schedule for laserpulses fired by the lidar transmitter can then be determined usingenergies that are predicted for the different scheduled laser pulseshots based on the laser energy model. This permits the lidartransmitter to reliably ensure at a highly granular level that eachlaser pulse shot has sufficient energy to meet operational needs,including when operating during periods of high density/high resolutionlaser pulse firing. The laser energy model is capable of modeling theenergy available for laser pulses in the laser source over very shorttime intervals as discussed in greater detail below. With such shortinterval time modeling, the laser energy modeling can be referred to asa transient laser energy model.

Furthermore, the inventors also disclose that mirror motion can bemodeled so that the system can also reliably predict where a scanningmirror is aimed within a field of view over time. This mirror motionmodel is also capable of predicting mirror motion over short timeintervals as discussed in greater detail below. In this regard, themirror motion model can also be referred to as a transient mirror motionmodel. The model of mirror motion over time can be linked with the modelof laser energy over time to provide still more granularity in thescheduling of laser pulses that are targeted at specific range points inthe field of view. Thus, a control circuit can translate a list ofarbitrarily ordered range points to be targeted with laser pulses into ashot list of laser pulses to be fired at such range points using themodeled laser energy coupled with the modeled mirror motion. In thisregard, the “shot list” can refer to a list of the range points to betargeted with laser pulses as combined with timing data that defines aschedule or sequence by which laser pulses will be fired toward suchrange points.

Through the use of such models, the lidar system can provide hypertemporal processing where laser pulses can be scheduled and fired athigh rates with high timing precision and high spatialtargeting/pointing precision. This results in a lidar system that canoperate at low latency, high frame rates, and intelligent range pointtargeting where regions of interest in the field of view can be targetedwith rapidly-fired and spatially dense laser pulse shots.

According to additional example embodiments, the inventors disclose thatthe detection intervals used by a lidar receiver to detect returns ofthe fired laser pulse shots can be closely controlled. Such control overthe detection intervals used by the lidar receiver allows for closecoordination between the lidar transmitter and the lidar receiver wherethe lidar receiver is able to adapt to variable shot intervals of thelidar transmitter (including periods of high rate firing as well asperiods of low rate firing).

Each detection interval can be associated with a different laser pulseshot from which a return is to be collected during the associateddetection interval. Accordingly, each detection interval is alsoassociated with the return for its associated laser pulse shot. Thelidar receiver can control these detection intervals on a shot-specificbasis so that the lidar receiver will be able to use the appropriatepixel sets for detecting the returns from the detection interval'sassociated shots. The lidar receiver includes a plurality of detectorpixels arranged as a photodetector array, and different sets of detectorpixels can be selected for use to detect the returns from differentlaser pulse shots. During a given detection interval, the lidar receiverwill collect sensed signal data from the selected pixel set, and thiscollected signal data can be processed to detect the associated returnfor that detection interval. The choice of which pixel set to use fordetecting a return from a given laser pulse shot can be based on thelocation in the field of the range point targeted by the given laserpulse shot. In this fashion, the lidar receiver will readout fromdifferent pixel sets during the detection intervals in a sequencedpattern that follows the sequenced spatial pattern of the laser pulseshots.

The lidar receiver can use any of a number of criteria for deciding whento start and stop reading out from the different pixel sets fordetecting returns. For example, the lidar receiver can use estimates ofpotential ranges to the targeted range points to decide on when thecollections should start and stop from various pixel sets. As anexample, if an object at range point X is located 10 meters from thelidar system, it can be expected that the return from the laser pulseshot fired at this object will reach the photodetector array relativelyquickly, while it would take relatively longer for a return to reach thephotodetector array if the object at range point X is located 1,000meters from the lidar system. To control when the collections shouldstart and stop from the pixel sets in order to detect returns from thelaser pulse shots, the system can determine pairs of minimum and maximumrange values for the range points targeted by each laser pulse shot, andthese minimum and maximum range values can be translated into on/offtimes for the pixel sets. Through intelligent control of these on (startcollection) and off (stop collection) times, the risk of missing areturn due to the return impacting a deactivated pixel is reduced.

Moreover, the detection intervals can vary across different shots (e.g.,Detection Interval A (associated with Shot A to support detection of thereturn from Shot A) can have a different duration than DetectionInterval B (associated with Shot B to support detection of the returnfrom Shot B)). Further still, at least some of the detection intervalscan be controlled to be of different durations than the shot intervalsthat correspond to such detection intervals. The shot interval thatcorresponds to a given detection interval is the time between the shotthat is associated with that detection interval and the next shot in theshot sequence. Counterintuitively, the inventors have found that it isoften not desirable for a detection interval to be of the same durationas its corresponding shot interval due to factors such as the amount ofprocessing time that is needed to detect returns within return signals.In many cases, it will be desirable for the control process to define adetection interval so that it exhibits a duration shorter than theduration of its corresponding shot interval; while in some other casesit may be desirable for the control process to define a detectioninterval so that it exhibits a longer duration than the duration of itscorresponding shot interval. This characteristic can be referred to as adetection interval that is asynchronous relative to its correspondingshot interval duration.

Further still, the inventors also disclose the use of multipleprocessors in a lidar receiver to distribute the workload of processingreturns. The activation/deactivation times of the pixel sets can be usedto define which samples in a return buffer will be used for processingto detect each return, and multiple processors can share the workload ofprocessing these samples in an effort to improve the latency of returndetection.

The inventors also disclose the use of multiple readout channels withina lidar receiver that are capable of simultaneously reading out sensedsignals from different pixel sets of the photodetector array. In doingso, the lidar receiver can support the use of overlapping detectionintervals when collecting signal data for detecting different returns.

Moreover, the inventors disclose a lidar system having a lidartransmitter and lidar receiver that are in a bistatic arrangement witheach other. Such a bistatic lidar system can be deployed in aclimate-controlled compartment of a vehicle to reduce the exposure ofthe lidar system to harsher elements so it can operate in moreadvantageous environments with regards to factors such as temperature,moisture, etc. In an example embodiment, the bistatic lidar system canbe connected to or incorporated within a rear view mirror assembly of avehicle.

These and other features and advantages of the invention will bedescribed in greater detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an example lidar transmitter that uses a laser energymodel to schedule laser pulses.

FIG. 2A depicts an example process flow the control circuit of FIG. 1.

FIG. 2B-2D depict additional examples of lidar transmitters that use alaser energy model to schedule laser pulses.

FIG. 3 depicts an example lidar transmitter that uses a laser energymodel and a mirror motion model to schedule laser pulses.

FIGS. 4A-4D illustrate how mirror motion can be modeled for a mirrorthat scans in a resonant mode.

FIG. 4E depicts an example process flow for controllably adjusting anamplitude for mirror scanning.

FIG. 5 depicts an example process flow for the control circuit of FIG.3.

FIGS. 6A and 6B depict example process flows for shot scheduling usingthe control circuit of FIG. 3.

FIG. 7A depicts an example process flow for simulating and evaluatingdifferent shot ordering candidates based on the laser energy model andthe mirror motion model.

FIG. 7B depicts an example of how time slots in a mirror scan can berelated to the shot angles for the mirror using the mirror motion model.

FIG. 7C depicts an example process flow for simulating different shotordering candidates based on the laser energy model.

FIGS. 7D-7F depict different examples of laser energy predictionsproduced by the laser energy model with respect to different shot ordercandidates.

FIG. 8 depicts an example lidar transmitter that uses a laser energymodel and a mirror motion model to schedule laser pulses, where thecontrol circuit includes a system controller and a beam scannercontroller.

FIG. 9 depicts an example process flow for inserting marker shots into ashot list.

FIG. 10 depicts an example process flow for using an eye safety model toadjust a shot list.

FIG. 11 depicts an example lidar transmitter that uses a laser energymodel, a mirror motion model, and an eye safety model to schedule laserpulses.

FIG. 12 depicts an example process flow for simulating different shotordering candidates based on the laser energy model and eye safetymodel.

FIG. 13 depicts another example process for determining shot schedulesusing the models.

FIG. 14 depicts an example lidar system where a lidar transmitter and alidar receiver coordinate their operations with each other.

FIG. 15 depicts another example process for determining shot schedulesusing the models.

FIG. 16 illustrates how the lidar transmitter can change its firing rateto probe regions in a field of view with denser groupings of laserpulses.

FIGS. 17A-17F depict example process flows for prioritized selections ofelevations with respect to shot scheduling.

FIG. 18A depicts an example lidar receiver in accordance with an exampleembodiment.

FIG. 18B depicts an example process flow for use by the lidar receiverof FIG. 18A to control the activation and deactivation of detectorpixels in a photodetector array.

FIG. 18C depicts an example lidar receiver in accordance with anotherexample embodiment.

FIG. 18D depicts an example process flow for use by the lidar receiverof FIG. 18C to control the activation and deactivation of detectorpixels in a photodetector array.

FIGS. 19A and 19B show examples of detection timing for a lidar receiverto detect returns from laser pulse shots.

FIG. 20 shows an example process flow for use by a signal processingcircuit of a lidar receiver to detect returns from laser pulse shots.

FIGS. 21A and 21B show examples of a multi-processor arrangement fordistributing the workload of detecting returns within a lidar receiver.

FIG. 22 shows an example process flow for assigning range swaths toreturn detections for a shot list of laser pulse shots.

FIGS. 23A and 23B show examples of mathematical operations that can beused to assign range swath values to return detections.

FIG. 24 shows another example process flow for assigning range swaths toreturn detections for a shot list of laser pulse shots.

FIG. 25 shows an example where a bistatic lidar system in accordancewith an example embodiment is deployed inside a climate-controlledcompartment of a vehicle.

FIG. 26 shows an example embodiment for a lidar receiver which employsmultiple readout channels to enable the use of overlapping detectionintervals for detecting the returns from different shots.

FIG. 27 shows an example embodiment of a lidar receiver that includesdetails showing how pixel activation can be controlled in concert withselective pixel readout.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG. 1 shows an example embodiment of a lidar transmitter 100 that canbe employed to support hyper temporal lidar. In an example embodiment,the lidar transmitter 100 can be deployed in a vehicle such as anautomobile. However, it should be understood that the lidar transmitter100 described herein need not be deployed in a vehicle. As used herein,“lidar”, which can also be referred to as “ladar”, refers to andencompasses any of light detection and ranging, laser radar, and laserdetection and ranging. In the example of FIG. 1, the lidar transmitter100 includes a laser source 102, a mirror subsystem 104, and a controlcircuit 106. Control circuit 106 uses a laser energy model 108 to governthe firing of laser pulses 122 by the laser source 102. Laser pulses 122transmitted by the laser source 102 are sent into the environment viamirror subsystem 104 to target various range points in a field of viewfor the lidar transmitter 100. These laser pulses 122 can beinterchangeably referred to as laser pulse shots (or more simply, asjust “shots”). The field of view will include different addressablecoordinates (e.g., {azimuth, elevation} pairs) which serve as rangepoints that can be targeted by the lidar transmitter 100 with the laserpulses 122.

In the example of FIG. 1, laser source 102 can use optical amplificationto generate the laser pulses 122 that are transmitted into the lidartransmitter's field of view via the mirror subsystem 104. In thisregard, a laser source 102 that includes an optical amplifier can bereferred to as an optical amplification laser source 102. In the exampleof FIG. 1, the optical amplification laser source 102 includes a seedlaser 114, an optical amplifier 116, and a pump laser 118. In this laserarchitecture, the seed laser 114 provides the input (signal) that isamplified to yield the transmitted laser pulse 122, while the pump laser118 provides the power (in the form of the energy deposited by the pumplaser 118 into the optical amplifier 116). So, the optical amplifier 116is fed by two inputs—the pump laser 118 (which deposits energy into theoptical amplifier 116) and the seed laser 114 (which provides the signalthat stimulates the energy in the optical amplifier 116 and inducespulse 122 to fire).

Thus, the pump laser 118, which can take the form of anelectrically-driven pump laser diode, continuously sends energy into theoptical amplifier 116. The seed laser 114, which can take the form of anelectrically-driven seed laser that includes a pulse formation networkcircuit, controls when the energy deposited by the pump laser 118 intothe optical amplifier 116 is released by the optical amplifier 116 as alaser pulse 122 for transmission. The seed laser 114 can also controlthe shape of laser pulse 122 via the pulse formation network circuit(which can drive the pump laser diode with the desired pulse shape). Theseed laser 114 also injects a small amount of (pulsed) optical energyinto the optical amplifier 116.

Given that the energy deposited in the optical amplifier 116 by the pumplaser 118 and seed laser 114 serves to seed the optical amplifier 116with energy from which the laser pulses 122 are generated, thisdeposited energy can be referred to as “seed energy” for the lasersource 102.

The optical amplifier 116 operates to generate laser pulse 122 from theenergy deposited therein by the seed laser 114 and pump laser 118 whenthe optical amplifier 116 is induced to fire the laser pulse 122 inresponse to stimulation of the energy therein by the seed laser 114. Theoptical amplifier 116 can take the form of a fiber amplifier. In such anembodiment, the laser source 102 can be referred to as a pulsed fiberlaser source. With a pulsed fiber laser source 102, the pump laser 118essentially places the dopant electrons in the fiber amplifier 116 intoan excited energy state. When it is time to fire laser pulse 122, theseed laser 114 stimulates these electrons, causing them to emit energyand collapse down to a lower (ground) state, which results in theemission of pulse 122. An example of a fiber amplifier that can be usedfor the optical amplifier 116 is a doped fiber amplifier such as anErbium-Doped Fiber Amplifier (EDFA).

It should be understood that other types of optical amplifiers can beused for the optical amplifier 116 if desired by a practitioner. Forexample, the optical amplifier 116 can take the form of a semiconductoramplifier. In contrast to a laser source that uses a fiber amplifier(where the fiber amplifier is optically pumped by pump laser 118), alaser source that uses a semiconductor amplifier can be electricallypumped. As another example, the optical amplifier 116 can take the formof a gas amplifier (e.g., a CO₂ gas amplifier). Moreover, it should beunderstood that a practitioner may choose to include a cascade ofoptical amplifiers 116 in laser source 102.

In an example embodiment, the pump laser 118 can exhibit a fixed rate ofenergy buildup (where a constant amount of energy is deposited in theoptical amplifier 116 per unit time). However, it should be understoodthat a practitioner may choose to employ a pump laser 118 that exhibitsa variable rate of energy buildup (where the amount of energy depositedin the optical amplifier 116 varies per unit time).

The laser source 102 fires laser pulses 122 in response to firingcommands 120 received from the control circuit 106. In an example wherethe laser source 102 is a pulsed fiber laser source, the firing commands120 can cause the seed laser 114 to induce pulse emissions by the fiberamplifier 116. In an example embodiment, the lidar transmitter 100employs non-steady state pulse transmissions, which means that therewill be variable timing between the commands 120 to fire the lasersource 102. In this fashion, the laser pulses 122 transmitted by thelidar transmitter 100 will be spaced in time at irregular intervals.There may be periods of relatively high densities of laser pulses 122and periods of relatively low densities of laser pulses 122. Examples oflaser vendors that provide such variable charge time control includeLuminbird and ITF. As examples, lasers that have the capacity toregulate pulse timing over timescales corresponding to preferredembodiments discussed herein and which are suitable to serve as lasersource 102 in these preferred embodiments are expected to exhibit laserwavelengths of 1.5 μm and available energies in a range of aroundhundreds of nano-Joules to around tens of micro-Joules, with timingcontrollable from hundreds of nanoseconds to tens of microseconds andwith an average power range from around 0.25 Watts to around 4 Watts.

The mirror subsystem 104 includes a mirror that is scannable to controlwhere the lidar transmitter 100 is aimed. In the example embodiment ofFIG. 1, the mirror subsystem 104 includes two mirrors—mirror 110 andmirror 112. Mirrors 110 and 112 can take the form of MEMS mirrors.However, it should be understood that a practitioner may choose toemploy different types of scannable mirrors. Mirror 110 is positionedoptically downstream from the laser source 102 and optically upstreamfrom mirror 112. In this fashion, a laser pulse 122 generated by thelaser source 102 will impact mirror 110, whereupon mirror 110 willreflect the pulse 122 onto mirror 112, whereupon mirror 112 will reflectthe pulse 122 for transmission into the environment. It should beunderstood that the outgoing pulse 122 may pass through varioustransmission optics during its propagation from mirror 112 into theenvironment.

In the example of FIG. 1, mirror 110 can scan through a plurality ofmirror scan angles to define where the lidar transmitter 100 is targetedalong a first axis. This first axis can be an X-axis so that mirror 110scans between azimuths. Mirror 112 can scan through a plurality ofmirror scan angles to define where the lidar transmitter 100 is targetedalong a second axis. The second axis can be orthogonal to the firstaxis, in which case the second axis can be a Y-axis so that mirror 112scans between elevations. The combination of mirror scan angles formirror 110 and mirror 112 will define a particular {azimuth, elevation}coordinate to which the lidar transmitter 100 is targeted. Theseazimuth, elevation pairs can be characterized as {azimuth angles,elevation angles} and/or {rows, columns} that define range points in thefield of view which can be targeted with laser pulses 122 by the lidartransmitter 100.

A practitioner may choose to control the scanning of mirrors 110 and 112using any of a number of scanning techniques. In a particularly powerfulembodiment, mirror 110 can be driven in a resonant mode according to asinusoidal signal while mirror 112 is driven in a point-to-point modeaccording to a step signal that varies as a function of the range pointsto be targeted with laser pulses 122 by the lidar transmitter 100. Inthis fashion, mirror 110 can be operated as a fast-axis mirror whilemirror 112 is operated as a slow-axis mirror. When operating in such aresonant mode, mirror 110 scans through scan angles in a sinusoidalpattern. In an example embodiment, mirror 110 can be scanned at afrequency in a range between around 100 Hz and around 20 kHz. In apreferred embodiment, mirror 110 can be scanned at a frequency in arange between around 10 kHz and around 15 kHz (e.g., around 12 kHz). Asnoted above, mirror 112 can be driven in a point-to-point mode accordingto a step signal that varies as a function of the range points to betargeted with laser pulses 122 by the lidar transmitter 100. Thus, ifthe lidar transmitter 100 is to fire a laser pulse 122 at a particularrange point having an elevation of X, then the step signal can drivemirror 112 to scan to the elevation of X. When the lidar transmitter 100is later to fire a laser pulse 122 at a particular range point having anelevation of Y, then the step signal can drive mirror 112 to scan to theelevation of Y. In this fashion, the mirror subsystem 104 canselectively target range points that are identified for targeting withlaser pulses 122. It is expected that mirror 112 will scan to newelevations at a much slower rate than mirror 110 will scan to newazimuths. As such, mirror 110 may scan back and forth at a particularelevation (e.g., left-to-right, right-to-left, and so on) several timesbefore mirror 112 scans to a new elevation. Thus, while the mirror 112is targeting a particular elevation angle, the lidar transmitter 100 mayfire a number of laser pulses 122 that target different azimuths at thatelevation while mirror 110 is scanning through different azimuth angles.U.S. Pat. Nos. 10,078,133 and 10,642,029, the entire disclosures ofwhich are incorporated herein by reference, describe examples of mirrorscan control using techniques and transmitter architectures such asthese (and others) which can be used in connection with the exampleembodiments described herein.

Control circuit 106 is arranged to coordinate the operation of the lasersource 102 and mirror subsystem 104 so that laser pulses 122 aretransmitted in a desired fashion. In this regard, the control circuit106 coordinates the firing commands 120 provided to laser source 102with the mirror control signal(s) 130 provided to the mirror subsystem104. In the example of FIG. 1, where the mirror subsystem 104 includesmirror 110 and mirror 112, the mirror control signal(s) 130 can includea first control signal that drives the scanning of mirror 110 and asecond control signal that drives the scanning of mirror 112. Any of themirror scan techniques discussed above can be used to control mirrors110 and 112. For example, mirror 110 can be driven with a sinusoidalsignal to scan mirror 110 in a resonant mode, and mirror 112 can bedriven with a step signal that varies as a function of the range pointsto be targeted with laser pulses 122 to scan mirror 112 in apoint-to-point mode.

As discussed in greater detail below, control circuit 106 can use alaser energy model 108 to determine a timing schedule for the laserpulses 122 to be transmitted from the laser source 102. This laserenergy model 108 can model the available energy within the laser source102 for producing laser pulses 122 over time in different shot schedulescenarios. By modeling laser energy in this fashion, the laser energymodel 108 helps the control circuit 106 make decisions on when the lasersource 102 should be triggered to fire laser pulses. Moreover, asdiscussed in greater detail below, the laser energy model 108 can modelthe available energy within the laser source 102 over short timeintervals (such as over time intervals in a range from 10-100nanoseconds), and such a short interval laser energy model 108 can bereferred to as a transient laser energy model 108.

Control circuit 106 can include a processor that provides thedecision-making functionality described herein. Such a processor cantake the form of a field programmable gate array (FPGA) orapplication-specific integrated circuit (ASIC) which providesparallelized hardware logic for implementing such decision-making. TheFPGA and/or ASIC (or other compute resource(s)) can be included as partof a system on a chip (SoC). However, it should be understood that otherarchitectures for control circuit 106 could be used, includingsoftware-based decision-making and/or hybrid architectures which employboth software-based and hardware-based decision-making. The processinglogic implemented by the control circuit 106 can be defined bymachine-readable code that is resident on a non-transitorymachine-readable storage medium such as memory within or available tothe control circuit 106. The code can take the form of software orfirmware that define the processing operations discussed herein for thecontrol circuit 106. This code can be downloaded onto the controlcircuit 106 using any of a number of techniques, such as a directdownload via a wired connection as well as over-the-air downloads viawireless networks, which may include secured wireless networks. As such,it should be understood that the lidar transmitter 100 can also includea network interface that is configured to receive such over-the-airdownloads and update the control circuit 106 with new software and/orfirmware. This can be particularly advantageous for adjusting the lidartransmitter 100 to changing regulatory environments with respect tocriteria such as laser dosage and the like. When using code provisionedfor over-the-air updates, the control circuit 106 can operate withunidirectional messaging to retain function safety.

Modeling Laser Energy Over Time:

FIG. 2A shows an example process flow for the control circuit 106 withrespect to using the laser energy model 108 to govern the timingschedule for laser pulses 122. At step 200, the control circuit 106maintains the laser energy model 108. This step can include reading theparameters and expressions that define the laser energy model 108,discussed in greater detail below. Step 200 can also include updatingthe laser energy model 108 over time as laser pulses 122 are triggeredby the laser source 102 as discussed below.

In an example embodiment where the laser source 102 is a pulsed fiberlaser source as discussed above, the laser energy model 108 can modelthe energy behavior of the seed laser 114, pump laser 118, and fiberamplifier 116 over time as laser pulses 122 are fired. As noted above,the fired laser pulses 122 can be referred to as “shots”. For example,the laser energy model 108 can be based on the following parameters:

-   -   CE(t), which represents the combined amount of energy within the        fiber amplifier 116 at the moment when the laser pulse 122 is        fired at time t.    -   EF(t), which represents the amount of energy fired in laser        pulse 122 at time t;    -   E_(P), which represents the amount of energy deposited by the        pump laser 118 into the fiber amplifier 116 per unit of time.    -   S(t+δ), which represents the cumulative amount of seed energy        that has been deposited by the pump laser 118 and seed laser 114        into the fiber amplifier 116 over the time duration δ, where δ        represents the amount of time between the most recent laser        pulse 122 (for firing at time t) and the next laser pulse 122        (to be fired at time t+δ).    -   F(t+δ), which represents the amount of energy left behind in the        fiber amplifier 116 when the pulse 122 is fired at time t (and        is thus available for use with the next pulse 122 to be fired at        time t+δ).    -   CE(t+δ), which represents the amount of combined energy within        the fiber amplifier 116 at time t+δ (which is the sum of S(t+δ)        and F(t+δ))    -   EF(t+δ), which represents the amount of energy fired in laser        pulse 122 fired at time t+δ    -   a and b, where “a” represents a proportion of energy transferred        from the fiber amplifier 116 into the laser pulse 122 when the        laser pulse 122 is fired, where “b” represents a proportion of        energy retained in the fiber amplifier 116 after the laser pulse        122 is fired, where a+b=1.

While the seed energy (S) includes both the energy deposited in thefiber amplifier 116 by the pump laser 118 and the energy deposited inthe fiber amplifier 116 by the seed laser 114, it should be understoodthat for most embodiments the energy from the seed laser 114 will bevery small relative to the energy from the pump laser 118. As such, apractitioner can choose to model the seed energy solely in terms ofenergy produced by the pump laser 118 over time.

Thus, after the pulsed fiber laser source 102 fires a laser pulse attime t, the pump laser 118 will begin re-supplying the fiber amplifier116 with energy over time (in accordance with E_(P)) until the seedlaser 116 is triggered at time t+δ to cause the fiber amplifier 116 toemit the next laser pulse 122 using the energy left over in the fiberamplifier 116 following the previous shot plus the new energy that hasbeen deposited in the fiber amplifier 116 by pump laser 118 since theprevious shot. As noted above, the parameters a and b model how much ofthe energy in the fiber amplifier 116 is transferred into the laserpulse 122 for transmission and how much of the energy is retained by thefiber amplifier 116 for use when generating the next laser pulse 122.

The energy behavior of pulsed fiber laser source 102 with respect to theenergy fired in laser pulses 122 in this regard can be expressed asfollows:

EF(t)=aCE(t)

F(t+δ)=bCE(t)

S(t+δ)=δE _(P)

CE(t+δ)=S(t+δ)+F(t+δ)

EF(t+δ)=aCE(t+δ)

With these relationships, the value for CE(t) can be re-expressed interms of EF(t) as follows:

${{CE}(t)} = \frac{{EF}(t)}{a}$

Furthermore, the value for F(t+δ) can be re-expressed in terms of EF(t)as follows:

${F\left( {t + \delta} \right)} = \frac{{bEF}(t)}{a}$

This means that the values for CE(t+δ) and EF(t+δ) can be re-expressedas follows:

${{{CE}\left( {t + \delta} \right)} = {{\delta E_{P}} + \frac{{bEF}(t)}{a}}}{{{EF}\left( {t + \delta} \right)} = {a\left( {{\delta E_{P}} + \frac{{bEF}(t)}{a}} \right)}}$

And this expression for EF(t+δ) shortens to:

EF(t+δ)=aδE _(P) +bEF(t)

It can be seen, therefore, that the energy to be fired in a laser pulse122 at time t+δ in the future can be computed as a function of how muchenergy was fired in the previous laser pulse 122 at time t. Given thata, b, E_(P), and EF(t) are known values, and δ is a controllablevariable, these expressions can be used as the laser energy model 108that predicts the amount of energy fired in a laser pulse at selecttimes in the future (as well as how much energy is present in the fiberamplifier 116 at select times in the future).

While this example models the energy behavior over time for a pulsedfiber laser source 102, it should be understood that these models couldbe adjusted to reflect the energy behavior over time for other types oflaser sources.

Thus, the control circuit 106 can use the laser energy model 108 tomodel how much energy is available in the laser source 102 over time andcan be delivered in the laser pulses 122 for different time schedules oflaser pulse shots. With reference to FIG. 2A, this allows the controlcircuit 106 to determine a timing schedule for the laser pulses 122(step 202). For example, at step 202, the control circuit 106 cancompare the laser energy model 108 with various defined energyrequirements to assess how the laser pulse shots should be timed. Asexamples, the defined energy requirements can take any of a number offorms, including but not limited to (1) a minimum laser pulse energy,(2) a maximum laser pulse energy, (3) a desired laser pulse energy(which can be per targeted range point for a lidar transmitter 100 thatselectively targets range points with laser pulses 122), (4) eye safetyenergy thresholds, and/or (5) camera safety energy thresholds. Thecontrol circuit 106 can then, at step 204, generate and provide firingcommands 120 to the laser source 102 that trigger the laser source 102to generate laser pulses 122 in accordance with the determined timingschedule. Thus, if the control circuit 106 determines that laser pulsesshould be generated at times t1, t2, t3, . . . , the firing commands 120can trigger the laser source to generate laser pulses 122 at thesetimes.

A control variable that the control circuit 106 can evaluate whendetermining the timing schedule for the laser pulses is the value of δ,which controls the time interval between successive laser pulse shots.The discussion below illustrates how the choice of δ impacts the amountof energy in each laser pulse 122 according to the laser energy model108.

For example, during a period where the laser source 102 is consistentlyfired every δ units of time, the laser energy model 108 can be used topredict energy levels for the laser pulses as shown in the following toyexample.

-   -   Toy Example 1, where E_(P)=1 unit of energy; δ=1 unit of time;        the initial amount of energy stored by the fiber laser 116 is 1        unit of energy; a=0.5 and b=0.5:

Shot Number 1 2 3 4 5 Time t + 1 t + 2 t + 3 t + 4 t + 5 Seed Energyfrom 1 1 1 1 1 Pump Laser (S) Leftover Fiber Energy (F) 1 1 1 1 1Combined Energy (S + F) 2 2 2 2 2 Energy Fired (EF) 1 1 1 1 1

If the rate of firing is increased, this will impact how much energy isincluded in the laser pulses. For example, relative to Toy Example 1, ifthe firing rate is doubled (δ=0.5 units of time) (while the otherparameters are the same), the laser energy model 108 will predict theenergy levels per laser pulse 122 as follows below with Toy Example 2.

-   -   Toy Example 2, where E_(P)=1 unit of energy; δ=0.5 units of        time; the initial amount of energy stored by the fiber laser 116        is 1 unit of energy; a=0.5 and b=0.5:

Shot Number 1 2 3 4 5 Time t + 0.5 t + 1 t + 1.5 t + 2 t + 3.5 SeedEnergy from 0.5 0.5 0.5 0.5 0.5 Pump Laser (S) Leftover Fiber 1 0.750.625 0.5625 0.53125 Energy (F) Combined Energy 1.5 1.25 1.125 1.06251.03125 (S + F) Energy Fired (EF) 0.75 0.625 0.5625 0.53125 0.515625

Thus, in comparing Toy Example 1 with Toy Example 2 it can be seen thatincreasing the firing rate of the laser will decrease the amount ofenergy in the laser pulses 122. As example embodiments, the laser energymodel 108 can be used to model a minimum time interval in a rangebetween around 10 nanoseconds to around 100 nanoseconds. This timing canbe affected by both the accuracy of the clock for control circuit 106(e.g., clock skew and clock jitter) and the minimum required refreshtime for the laser source 102 after firing.

If the rate of firing is decreased relative to Toy Example 1, this willincrease how much energy is included in the laser pulses. For example,relative to Toy Example 1, if the firing rate is halved (δ=2 units oftime) (while the other parameters are the same), the laser energy model108 will predict the energy levels per laser pulse 122 as follows belowwith Toy Example 3.

-   -   Toy Example 3, where E_(P)=1 unit of energy; δ=2 units of time;        the initial amount of energy stored by the fiber laser 116 is 1        unit of energy; a=0.5 and b=0.5:

Shot Number 1 2 3 4 5 Time t + 2 t + 4 t + 6 t + 8 t + 10 Seed Energyfrom 2 2 2 2 2 Pump Laser (S) Leftover Fiber 1 1.5 1.75 1.875 1.9375Energy (F) Combined Energy 3 3.5 3.75 3.875 3.9375 (S + F) Energy Fired(EF) 1.5 1.75 1.875 1.9375 1.96875

If a practitioner wants to maintain a consistent amount of energy perlaser pulse, it can be seen that the control circuit 106 can use thelaser energy model 108 to define a timing schedule for laser pulses 122that will achieve this goal (through appropriate selection of values forδ).

For practitioners that want the lidar transmitter 100 to transmit laserpulses at varying intervals, the control circuit 106 can use the laserenergy model 108 to define a timing schedule for laser pulses 122 thatwill maintain a sufficient amount of energy per laser pulse 122 in viewof defined energy requirements relating to the laser pulses 122. Forexample, if the practitioner wants the lidar transmitter 100 to have theability to rapidly fire a sequence of laser pulses (for example, tointerrogate a target in the field of view with high resolution) whileensuring that the laser pulses in this sequence are each at or abovesome defined energy minimum, the control circuit 106 can define a timingschedule that permits such shot clustering by introducing a sufficientlylong value for δ just before firing the clustered sequence. This long δvalue will introduce a “quiet” period for the laser source 102 thatallows the energy in seed laser 114 to build up so that there issufficient available energy in the laser source 102 for the subsequentrapid fire sequence of laser pulses. As indicated by the decay patternof laser pulse energy reflected by Toy Example 2, increasing thestarting value for the seed energy (S) before entering the time periodof rapidly-fired laser pulses will make more energy available for thelaser pulses fired close in time with each other.

Toy Example 4 below shows an example shot sequence in this regard, wherethere is a desire to fire a sequence of 5 rapid laser pulses separatedby 0.25 units of time, where each laser pulse has a minimum energyrequirement of 1 unit of energy. If the laser source has just concludeda shot sequence after which time there is 1 unit of energy retained inthe fiber laser 116, the control circuit can wait 25 units of time toallow sufficient energy to build up in the seed laser 114 to achieve thedesired rapid fire sequence of 5 laser pulses 122, as reflected in thetable below.

-   -   Toy Example 4, where E_(P)=1 unit of energy; δ_(LONG)=25 units        of time; δ_(SHORT)=0.25 units of time; the initial amount of        energy stored by the fiber laser 116 is 1 unit of energy; a=0.5        and b=0.5; and the minimum pulse energy requirement is 1 unit of        energy:

Shot Number 1 2 3 4 5 Time t + 25 t + 25.25 t + 25.5 t + 25.75 t + 26Seed Energy from 25 0.25 0.25 0.25 0.25 Pump Laser (S) Leftover Fiber 113 6.625 3.4375 1.84375 Energy (F) Combined Energy 26 13.25 6,875 3.68752.09375 (S + F) Energy Fired (EF) 13 6.625 3.4375 1.84375 1.046875

This ability to leverage “quiet” periods to facilitate “busy” periods oflaser activity means that the control circuit 106 can provide highlyagile and responsive adaptation to changing circumstances in the fieldof view. For example, FIG. 16 shows an example where, during a firstscan 1600 across azimuths from left to right at a given elevation, thelaser source 102 fires 5 laser pulses 122 that are relatively evenlyspaced in time (where the laser pulses are denoted by the “X” marks onthe scan 1600). If a determination is made that an object of interest isfound at range point 1602, the control circuit 106 can operate tointerrogate the region of interest 1604 around range point 1602 with ahigher density of laser pulses on second scan 1610 across the azimuthsfrom right to left. To facilitate this high density period of rapidlyfired laser pulses within the region of interest 1604, the controlcircuit 106 can use the laser energy model 108 to determine that suchhigh density probing can be achieved by inserting a lower density period1606 of laser pulses during the time period immediately prior toscanning through the region of interest 1604. In the example of FIG. 16,this lower density period 1604 can be a quiet period where no laserpulses are fired. Such timing schedules of laser pulses can be definedfor different elevations of the scan pattern to permit high resolutionprobing of regions of interest that are detected in the field of view.

The control circuit 106 can also use the energy model 108 to ensure thatthe laser source 102 does not build up with too much energy. Forpractitioners that expect the lidar transmitter 100 to exhibit periodsof relatively infrequent laser pulse firings, it may be the case thatthe value for δ in some instances will be sufficiently long that toomuch energy will build up in the fiber amplifier 116, which can causeproblems for the laser source 102 (either due to equilibrium overheatingof the fiber amplifier 116 or non-equilibrium overheating of the fiberamplifier 116 when the seed laser 114 induces a large amount of pulseenergy to exit the fiber amplifier 116). To address this problem, thecontrol circuit 106 can insert “marker” shots that serve to bleed offenergy from the laser source 102. Thus, even though the lidartransmitter 100 may be primarily operating by transmitting laser pulses122 at specific, selected range points, these marker shots can be firedregardless of the selected list of range points to be targeted for thepurpose of preventing damage to the laser source 102. For example, ifthere is a maximum energy threshold for the laser source 102 of 25 unitsof energy, the control circuit 106 can consult the laser energy model108 to identify time periods where this maximum energy threshold wouldbe violated. When the control circuit 106 predicts that the maximumenergy threshold would be violated because the laser pulses have beentoo infrequent, the control circuit 106 can provide a firing command 120to the laser source 102 before the maximum energy threshold has beenpassed, which triggers the laser source 102 to fire the marker shot thatbleeds energy out of the laser source 102 before the laser source'senergy has gotten too high. This maximum energy threshold can be trackedand assessed in any of a number of ways depending on how the laserenergy model 108 models the various aspects of laser operation. Forexample, it can be evaluated as a maximum energy threshold for the fiberamplifier 116 if the energy model 108 tracks the energy in the fiberamplifier 116 (S+F) over time. As another example, the maximum energythreshold can be evaluated as a maximum value of the duration δ (whichwould be set to prevent an amount of seed energy (S) from beingdeposited into the fiber amplifier 116 that may cause damage when takingthe values for E_(P) and a presumed value for F into consideration.

While the toy examples above use simplified values for the modelparameters (e.g. the values for E_(P) and δ) for the purpose of ease ofexplanation, it should be understood that practitioners can selectvalues for the model parameters or otherwise adjust the model componentsto accurately reflect the characteristics and capabilities of the lasersource 102 being used. For example, the values for E_(P), a, and b canbe empirically determined from testing of a pulsed fiber laser source(or these values can be provided by a vendor of the pulsed fiber lasersource). Moreover, a minimum value for δ can also be a function of thepulsed fiber laser source 102. That is, the pulsed fiber laser sourcesavailable from different vendors may exhibit different minimum valuesfor δ, and this minimum value for δ (which reflects a maximum achievablenumber of shots per second) can be included among the vendor'sspecifications for its pulsed fiber laser source.

Furthermore, in situations where the pulsed fiber laser source 102 isexpected or observed to exhibit nonlinear behaviors, such nonlinearbehavior can be reflected in the model. As an example, it can beexpected that the pulsed fiber laser source 102 will exhibit energyinefficiencies at high power levels. In such a case, the modeling of theseed energy (S) can make use of a clipped, offset (affine) model for theenergy that gets delivered to the fiber amplifier 116 by pump laser 118for pulse generation. For example, in this case, the seed energy can bemodeled in the laser energy model 108 as:

S(t+δ)=E _(P)max(a ₁ δ+a ₀, offset)

The values for a₁, a₀, and offset can be empirically measured for thepulsed fiber laser source 102 and incorporated into the modeling ofS(t+δ) used within the laser energy model 108. It can be seen that for alinear regime, the value for a₁ would be 1, and the values for a₀ andoffset would be 0. In this case, the model for the seed energy S(t+δ)reduces to δE_(P) as discussed in the examples above.

The control circuit 106 can also update the laser energy model 108 basedon feedback that reflects the energies within the actual laser pulses122. In this fashion, laser energy model 108 can better improve ormaintain its accuracy over time. In an example embodiment, the lasersource 102 can monitor the energy within laser pulses 122 at the time offiring. This energy amount can then be reported by the laser source 102to the control circuit 106 (see 250 in FIG. 2B) for use in updating themodel 108. Thus, if the control circuit 106 detects an error between theactual laser pulse energy and the modeled pulse energy, then the controlcircuit 106 can introduce an offset or other adjustment into model 108to account for this error.

For example, it may be necessary to update the values for a and b toreflect actual operational characteristics of the laser source 102. Asnoted above, the values of a and b define how much energy is transferredfrom the fiber amplifier 116 into the laser pulse 122 when the lasersource 102 is triggered and the seed laser 114 induces the pulse 122 toexit the fiber amplifier 116. An updated value for a can be computedfrom the monitored energies in transmitted pulses 122 (PE) as follows:

a=argmin_(a)(Σ_(k=1 . . . N) |PE(t _(k)+δ_(k))−aPE(t _(k))−(1−a)δt_(k)|²)

In this expression, the values for PE represent the actual pulseenergies at the referenced times (t_(k) or t_(k)+δ_(k)). This is aregression problem and can be solved using commercial software toolssuch as those available from MATLAB, Wolfram, PTC, ANSYS, and others. Inan ideal world, the respective values for PE(t) and PE(t+δ) will be thesame as the modeled values of EF(t) and EF(t+δ). However, for a varietyof reasons, the gain factors a and b may vary due to laser efficiencyconsiderations (such as heat or aging whereby back reflections reducethe resonant efficiency in the laser cavity). Accordingly, apractitioner may find it useful to update the model 108 over time toreflect the actual operational characteristics of the laser source 102by periodically computing updated values to use for a and b.

In scenarios where the laser source 102 does not report its own actuallaser pulse energies, a practitioner can choose to include aphotodetector at or near an optical exit aperture of the lidartransmitter 100 (e.g., see photodetector 252 in FIG. 2C). Thephotodetector 252 can be used to measure the energy within thetransmitted laser pulses 122 (while allowing laser pulses 122 topropagate into the environment toward their targets), and these measuredenergy levels can be used to detect potential errors with respect to themodeled energies for the laser pulses so model 108 can be adjusted asnoted above. As another example for use in a scenario where the lasersource 102 does not report its own actual laser pulse energies, apractitioner derives laser pulse energy from return data 254 withrespect to returns from known fiducial objects in a field of view (suchas road signs which are regulated in terms of their intensity values forlight returns) (see 254 in FIG. 2D) as obtained from a point cloud 256for the lidar system. Additional details about such energy derivationsare discussed below. Thus, in such an example, the model 108 can beperiodically re-calibrated using point cloud data for returns from suchfiducials, whereby the control circuit 106 derives the laser pulseenergy that would have produced the pulse return data found in the pointcloud 256. This derived amount of laser pulse energy can then becompared with the modeled laser pulse energy for adjustment of the laserenergy model 108 as noted above.

Modeling Mirror Motion Over Time:

In a particularly powerful example embodiment, the control circuit 106can also model mirror motion to predict where the mirror subsystem 104will be aimed at a given point in time. This can be especially helpfulfor lidar transmitters 100 that selectively target specific range pointsin the field of view with laser pulses 122. By coupling the modeling oflaser energy with a model of mirror motion, the control circuit 106 canset the order of specific laser pulse shots to be fired to targetedrange points with highly granular and optimized time scales. Asdiscussed in greater detail below, the mirror motion model can modelmirror motion over short time intervals (such as over time intervals ina range from 5-50 nanoseconds). Such a short interval mirror motionmodel can be referred to as a transient mirror motion model.

FIG. 3 shows an example lidar transmitter 100 where the control circuit106 uses both a laser energy model 108 and a mirror motion model 308 togovern the timing schedule for laser pulses 122.

In an example embodiment, the mirror subsystem 104 can operate asdiscussed above in connection with FIG. 1. For example, the controlcircuit 106 can (1) drive mirror 110 in a resonant mode using asinusoidal signal to scan mirror 110 across different azimuth angles and(2) drive mirror 112 in a point-to-point mode using a step signal toscan mirror 112 across different elevations, where the step signal willvary as a function of the elevations of the range points to be targetedwith laser pulses 122. Mirror 110 can be scanned as a fast-axis mirror,while mirror 112 is scanned as a slow-axis mirror. In such anembodiment, a practitioner can choose to use the mirror motion model 308to model the motion of mirror 110 as (comparatively) mirror 112 can becharacterized as effectively static for one or more scans across azimuthangles.

FIGS. 4A-4C illustrate how the motion of mirror 110 can be modeled overtime. In these examples, (1) the angle theta (θ) represents the tiltangle of mirror 110, (2) the angle phi (Φ) represents the angle at whicha laser pulse 122 from the laser source 102 will be incident on mirror110 when mirror 110 is in a horizontal position (where θ is zerodegrees—see FIG. 4A), and (3) the angle mu (μ) represents the angle ofpulse 422 as reflected by mirror 110 relative to the horizontal positionof mirror 110. In this example, the angle μ can represent the scan angleof the mirror 110, where this scan angle can also be referred to as ashot angle for mirror 110 as angle μ corresponds to the angle at whichreflected laser pulse 122′ will be directed into the field of view iffired at that time.

FIG. 4A shows mirror 110, where mirror 110 is at “rest” with a tiltangle θ of zero degrees, which can be characterized as the horizon ofmirror 110. Laser source 102 is oriented in a fixed position so thatlaser pulses 122 will impact mirror 110 at the angle Φ relative to thehorizontal position of mirror 110. Given the property of reflections, itshould be understood that the value of the shot angle μ will be the sameas the value of angle Φ when the mirror 110 is horizontal (where θ=0).

FIG. 4B shows mirror 110 when it has been tilted about pivot 402 to apositive non-zero value of θ. It can be seen that the tilting of mirrorto angle θ will have the effect of steering the reflected laser pulse122′ clockwise and to the right relative to the angle of the reflectedlaser pulse 122′ in FIG. 4A (when mirror 110 was horizontal).

Mirror 110 will have a maximum tilt angle that can be referred to as theamplitude A of mirror 110. Thus, it can be understood that mirror 110will scan through its tilt angles between the values of −A (whichcorresponds to −θ_(Max)) and +A (which corresponds to +θ_(Max)). It canbe seen that the angle of reflection for the reflected laser pulse 122′relative to the actual position of mirror 110 is the sum of θ+Φ as shownby FIG. 4B. In then follows that the value of the shot angle μ will beequal to 2θ+Φ, as can be seen from FIG. 4B.

When driven in a resonant mode according to sinusoidal control signal,mirror 110 will change its tilt angle θ according to a cosineoscillation, where its rate of change is slowest at the ends of its scan(when it changes its direction of tilt) and fastest at the mid-point ofits scan. In an example where the mirror 110 scans between maximum tiltangles of −A to +A, the value of the angle θ as a function of time canbe expressed as:

θ=A cos(2πft)

where f represents the scan frequency of mirror 110 and t representstime. Based on this model, it can be seen that the value for θ can varyfrom A (when t=0) to 0 (when t is a value corresponding to 90 degrees ofphase (or 270 degrees of phase) to −A (when t is a value correspondingto 180 degrees of phase).

This means that the value of the shot angle μ can be expressed as afunction of time by substituting the cosine expression for θ into theexpression for the shot angle of μ=2θ+Φ as follows:

μ=2A cos(2πft)+φ

From this expression, one can then solve for t to produce an expressionas follows:

$t = \frac{\arccos\left( \frac{\mu - \varphi}{2A} \right)}{2\pi f}$

This expression thus identifies the time t at which the scan of mirror110 will target a given shot angle μ. Thus, when the control circuit 106wants to target a shot angle of μ, the time at which mirror 110 willscan to this shot angle can be readily computed given that the valuesfor Φ, A, and f will be known. In this fashion, the mirror motion model308 can model that shot angle as a function of time and predict the timeat which the mirror 110 will target a particular shot angle.

FIG. 4C shows mirror 110 when it has been tilted about pivot 402 to anegative non-zero value of −θ. It can be seen that the tilting of mirrorto angle −θ will have the effect of steering the reflected laser pulse122′ counterclockwise and to the left relative to the angle of thereflected laser pulse 122′ in FIG. 4A (when mirror 110 was horizontal).FIG. 4C also demonstrates a constraint for a practitioner on theselection of the value for the angle Φ. Laser source 102 will need to bepositioned so that the angle Φ is greater than the value of A to avoid asituation where the underside of the tilted mirror 110 occludes thelaser pulse 122 when mirror is tilted to a value of θ that is greaterthan Φ. Furthermore, the value of the angle Φ should not be 90° to avoida situation where the mirror 110 will reflect the laser pulse 122 backinto the laser source 102. A practitioner can thus position the lasersource 102 at a suitable angle Φ accordingly.

FIG. 4D illustrates a translation of this relationship to how the mirror110 scans across a field of view 450. The mirror 110 will alternatelyscan in a left-to-right direction 452 and right-to-left direction 454 asmirror 110 tilts between its range of tilt angles (e.g., θ=−A through+A). For the example of FIG. 4A where the value for θ is zero, thismeans that a laser pulse fired at the untilted mirror 110 will bedirected as shown by 460 in FIG. 4D, where the laser pulse is directedtoward a range point at the mid-point of scan. The shot angle μ for this“straight ahead” gaze is Φ as discussed above in connection with FIG.4A. As the angle θ increases from θ=0, this will cause the laser pulsesdirected by mirror 110 to scan to the right in the field of view untilthe mirror 110 tilts to the angle θ=+A. When θ=+A, mirror 110 will be atthe furthest extent of its rightward scan 452, and it will direct alaser pulse as shown by 462. The shot angle μ for this rightmost scanposition will be the value μ=2A+Φ. From that point, the mirror 110 willbegin scanning leftward in direction 454 by reducing its tilt angle θ.The mirror 110 will once again scan through the mid-point and eventuallyreach a tilt angle of θ=−A. When θ=−A, mirror 110 will be at thefurthest extent of its leftward scan 452, and it will direct a laserpulse as shown by 464. The shot angle μ for this leftmost scan positionwill be the value μ=Φ−2A. From that point, the mirror 110 will begintilting in the rightward direction 450 again, and the scan repeats. Asnoted above, due to the mirror motion model 308, the control circuit 106will know the time at which the mirror 110 is targeting a shot angle ofμ_(i) to direct a laser pulse as shown by 466 of FIG. 4D.

In an example embodiment, the values for +A and −A can be values in arange between +/−10 degrees and +/−20 degrees (e.g., +/−16 degrees)depending on the nature of mirror chosen as mirror 110. In an examplewhere A is 16 degrees and mirror 110 scans as discussed above inconnection with FIGS. 4A-4D, it can be understood that the angularextent of the scan for mirror 110 would be 64 degrees (or 2A from thescan mid-point in both the right and left directions for a total of 4A).

In some example embodiments, the value for A in the mirror motion model308 can be a constant value. However, some practitioners may find itdesirable to deploy a mirror 110 that exhibits an adjustable value for A(e.g., a variable amplitude mirror such as a variable amplitude MEMSmirror can serve as mirror 110). From the relationships discussed above,it can be seen that the time required to move between two shot angles isreduced when the value for amplitude A is reduced. The control circuit106 can leverage this relationship to determine whether it is desirableto adjust the amplitude of the mirror 110 before firing a sequence oflaser pulses 122. FIG. 4E shows an example process flow in this regard.At step 470, the control circuit 106 determines the settle time (ts) forchanging the amplitude from A to A′ (where A′<A). It should beunderstood that changing the mirror amplitude in this fashion willintroduce a time period where the mirror is relatively unstable, andtime will need to be provided to allow the mirror to settle down to astable position. This settling time can be empirically determined ortracked for the mirror 110, and the control circuit 106 can maintainthis settle time value as a control parameter. At step 472, the controlcircuit 106 determines the time it will take to collect a shot list dataset in a circumstance where the amplitude of the mirror is unchanged(amplitude remains A). This time can be referenced as collection timetc. This value for tc can be computed through the use of the laserenergy model 108 and mirror motion model 308 with reference to the shotsincluded in a subject shot list. At step 474, the control circuit 106determines the time it will take to collect the same shot list data setin a circumstance where the amplitude of the mirror is changed to A′.This time can be referenced as collection time tc′. This value for tc′can be computed through the use of the laser energy model 108 and mirrormotion model 308 (as adjusted in view of the reduced amplitude of A′)with reference to the shots included in the subject shot list. At step476, the control circuit compares tc with the sum of tc′ and ts. If thesum (tc′+ts) is less than tc, this means that it will be time efficientto change the mirror amplitude to A′. In this circumstance, the processflow proceeds to step 478, and the control circuit 106 adjusts theamplitude of mirror 110 to A′. If the sum (tc′+ts) is not less than tc,then the control circuit 106 leaves the amplitude value unchanged (step480).

Model-Based Shot Scheduling:

FIG. 5 shows an example process flow for the control circuit 106 to useboth the laser energy model 108 and the mirror motion model 308 todetermine the timing schedule for laser pulses 122. Step 200 can operateas described above with reference to FIG. 2A to maintain the laserenergy model 108. At step 500, the control circuit 106 maintains themirror motion model 308. As discussed above, this model 308 can modelthe shot angle that the mirror will target as a function of time.Accordingly, the mirror motion model 308 can predict the shot angle ofmirror 110 at a given time t. To maintain and update the model 308, thecontrol circuit 108 can establish the values for A, Φ, and f to be usedfor the model 308. These values can be read from memory or determinedfrom the operating parameters for the system.

At step 502, the control circuit 106 determines a timing schedule forlaser pulses 122 using the laser energy model 108 and the mirror motionmodel 308. By linking the laser energy model 108 and the mirror motionmodel 308 in this regard, the control circuit 106 can determine how muchenergy is available for laser pulses targeted toward any of the rangepoints in the scan pattern of mirror subsystem 104. For purposes ofdiscussion, we will consider an example embodiment where mirror 110scans in azimuth between a plurality of shot angles at a high rate whilemirror 112 scans in elevation at a sufficiently slower rate so that thediscussion below will assume that the elevation is held steady whilemirror 110 scans back and forth in azimuth. However, the techniquesdescribed herein can be readily extended to modeling the motion of bothmirrors 110 and 112.

If there is a desire to target a range point at a Shot Angle A with alaser pulse of at least X units of energy, the control circuit 106, atstep 502, can consult the laser energy model 108 to determine whetherthere is sufficient laser energy for the laser pulse when the mirror110's scan angle points at Shot Angle A. If there is sufficient energy,the laser pulse 122 can be fired when the mirror 110 scans to Shot AngleA. If there is insufficient energy, the control circuit 106 can wait totake the shot until after mirror 110 has scanned through and back topointing at Shot Angle A (if the laser energy model 108 indicates thereis sufficient laser energy when the mirror returns to Shot Angle A). Thecontrol circuit 106 can compare the shot energy requirements for a setof shot angles to be targeted with laser pulses to determine when thelaser pulses 122 should be fired. Upon determination of the timingschedule for the laser pulses 122, the control circuit 106 can generateand provide firing commands 120 to the laser source 102 based on thisdetermined timing schedule (step 504).

FIGS. 6A and 6B show example process flows for implementing steps 502and 504 of FIG. 5 in a scenario where the mirror subsystem 104 includesmirror 110 that scans through azimuth shot angles in a resonant mode(fast-axis) and mirror 112 that scans through elevation shot angles in apoint-to-point mode (slow-axis). Lidar transmitter 100 in these examplesseeks to fire laser pulses 122 at intelligently selected range points inthe field of view. With the example of FIG. 6A, the control circuit 106schedules shots for batches of range points at a given elevation onwhichever scan direction of the mirror 110 is schedulable for thoserange points according to the laser energy model 108. With the exampleof FIG. 6B, the control circuit 106 seeks to schedule shots for as manyrange points as it can at a given elevation for each scan direction ofthe mirror 110 in view of the laser energy model 108. For any shots atthe subject elevation that cannot be scheduled for a given scandirection due to energy model constraints, the control circuit 106 thenseeks to schedule those range points on the reverse scan (and so onuntil all of the shots are scheduled).

The process flow of FIG. 6A begins with step 600. At step 600, thecontrol circuit 106 receives a list of range points to be targeted withlaser pulses. These range points can be expressed as (azimuth angle,elevation angle) pairs, and they may be ordered arbitrarily.

At step 602, the control circuit 106 sorts the range points by elevationto yield sets of azimuth shot angles sorted by elevation. Theelevation-sorted range points can also be sorted by azimuth shot angle(e.g., where all of the shot angles at a given elevation are sorted inorder of increasing azimuth angle (smallest azimuth shot angle tolargest azimuth shot angle) or decreasing azimuth angle (largest azimuthshot angle to smallest azimuth shot angle). For the purposes ofdiscussing the process flows of FIGS. 6A and 6B, these azimuth shotangles can be referred to as the shot angles for the control circuit106. Step 602 produces a pool 650 of range points to be targeted withshots (sorted by elevation and then by shot angle).

At step 604, the control circuit 106 selects a shot elevation from amongthe shot elevations in the sorted list of range points in pool 650. Thecontrol circuit 106 can make this selection on the basis of any of anumber of criteria. The order of selection of the elevations will governwhich elevations are targeted with laser pulses 122 before others.

Accordingly, in an example embodiment, the control circuit 106 canprioritize the selection of elevations at step 604 that are expected toencompass regions of interest in the field of view. As an example, somepractitioners may find the horizon in the field of view (e.g., a roadhorizon) to be high priority for targeting with laser pulses 122. Insuch a case, step 604 can operate as shown by FIG. 17A to determine theelevation(s) which correspond to a horizon in the field of view (e.g.identify the elevations at or near the road horizon) (see step 1702) andthen prioritize the selection of those elevations from pool 650 (seestep 1702). Step 1702 can be performed by analyzing lidar return pointcloud data and/or camera images of the field of view to identify regionsin the field of view that are believed to qualify as the horizon (e.g.,using contrast detection techniques, edge detection techniques, and/orother pattern processing techniques applied to lidar or image data).

As another example, the control circuit 106 can prioritize the selectionof elevations based on the range(s) to detected object(s) in the fieldof view. Some practitioners may find it desirable to prioritize theshooting of faraway objects in the field of view. Other practitionersmay find it desirable to prioritize the shooting of nearby objects inthe field of view. Thus, in an example such as that shown by FIG. 17B,the range(s) applicable to detected object(s) is determined (see step1706). This range information will be available from the lidar returnpoint cloud data. At step 1708, the control circuit sorts the detectedobject(s) by their determined range(s). Then, at step 1708, the controlcircuit 106 prioritizes the selection of elevations from pool 650 basedon the determined range(s) for object(s) included in those elevations.With step 1708, prioritization can be given to larger range values thanfor smaller range values if the practitioner wants to shoot farawayobjects before nearby objects. For practitioners that want to shootnearby objects before faraway objects, step 1708 can give priority tosmaller range values than for larger range values. Which objects aredeemed faraway and which are deemed nearby can be controlled using anyof a number of techniques. For example, a range threshold can bedefined, and the control circuit 106 can make the elevation selectionsbased on which elevations include sorted objects whose range is above(or below as the case may be) the defined range threshold. As anotherexample, the relative ranges for the sorted objects can be used tocontrol the selection of elevations (where the sort order of eitherfarthest to nearest or nearest to farthest governs the selection ofelevations which include those objects).

As yet another example, the control circuit 106 can prioritize theselection of elevations based on the velocity(ies) of detected object(s)in the field of view. Some practitioners may find it desirable toprioritize the shooting of fast-moving objects in the field of view.FIG. 17C shows an example process flow for this. At step 1714, thevelocity is determined for each detected object in the field of view.This velocity information can be derived from the lidar return pointcloud data. At step 1716, the control circuit 106 can sort the detectedobject(s) by the determined velocity(ies). The control circuit 106 canthen use determined velocities for the sorted objects as a basis forprioritizing the selection of elevations which contain those detectedobjects (step 1718). This prioritization at step 1718 can be carried outin any of a number of ways. For example, a velocity threshold can bedefined, and step 1718 can prioritize the selection of elevation includean object moving at or above this defined velocity threshold. As anotherexample, the relative velocities of the sorted objects can be used wherean elevation that includes an object moving faster than another objectcan be selected before an elevation that includes the another (slowermoving) object.

As yet another example, the control circuit 106 can prioritize theselection of elevations based on the directional heading(s) of detectedobject(s) in the field of view. Some practitioners may find it desirableto prioritize the shooting of objects in the field of view that movingtoward the lidar transmitter 100. FIG. 17D shows an example process flowfor this. At step 1720, the directional heading is determined for eachdetected object in the field of view. This directional heading can bederived from the lidar return point cloud data. The control circuit 1722can then prioritize the selection of elevation(s) that include object(s)that are determined to be heading toward the lidar transmitter 100(within some specified degree of tolerance where the elevation thatcontains an object heading near the lidar transmitter 100 would beselected before an elevation that contains an object moving away fromthe lidar transmitter 100).

Further still, some practitioners may find it desirable to combine theprocess flows of FIGS. 17C and 17D to prioritize the selection offast-moving objects that are heading toward the lidar transmitter 100.An example for this is shown by FIG. 17E. With FIG. 17E, steps 1714 and1720 can be performed as discussed above. At step 1724, the detectedobject(s) are sorted by their directional headings (relative to thelidar transmitter 100) and then by the determined velocities. At step1726, the elevations which contain objected deemed to be heading towardthe lidar transmitter 100 (and moving faster than other such objects)are prioritized for selection.

In another example embodiment, the control circuit 106 can selectelevations at step 604 based on eye safety or camera safety criteria.For example, eye safety requirements may specify that the lidartransmitter 100 should not direct more than a specified amount of energyin a specified spatial area over of a specified time period. To reducethe risk of firing too much energy into the specified spatial area, thecontrol circuit 106 can select elevations in a manner that avoidssuccessive selections of adjacent elevations (e.g., jumping fromElevation 1 to Elevation 3 rather than Elevation 2) to insert moreelevation separation between laser pulses that may be fired close intime. This manner of elevation selection may optionally be implementeddynamically (e.g., where elevation skips are introduced if the controlcircuit 106 determines that the energy in a defined spatial area hasexceeded some level that is below but approaching the eye safetythresholds). Furthermore, it should be understood that the number ofelevations to skip (a skip interval) can be a value selected by apractitioner or user to define how many elevations will be skipped whenprogressing from elevation-to-elevation. As such, a practitioner maychoose to set the elevation skip interval to be a value larger than 1(e.g., a skip interval of 5, which would cause the system to progressfrom Elevation 3 to Elevation 9). Furthermore, similar measures can betaken to avoid hitting cameras that may be located in the field of viewwith too much energy. FIG. 17F depicts an example process flow for thisapproach. At step 1730, the control circuit 106 selects Elevation X_(t)(where this selected elevation is larger (or smaller) than the precedingselected elevation (Elevation X_(t−1)) by the defined skip interval.Then, the control circuit 106 schedules the shots for the selectedelevation (step 1732), and the process flow returns to step 1730 wherethe next elevation (Elevation X_(t+1)) is selected (according to theskip interval relative to Elevation X_(t)).

Thus, it should be understood that step 604 can employ a prioritizedclassification system that decides the order in which elevations are tobe targeted with laser pulses 122 based on the criteria of FIGS. 17A-17For any combinations of any of these criteria.

At step 606, the control circuit 106 generates a mirror control signalfor mirror 112 to drive mirror 112 so that it targets the angle of theselected elevation. As noted, this mirror control signal can be a stepsignal that steps mirror 112 up (or down) to the desired elevationangle. In this fashion, it can be understood that the control circuit106 will be driving mirror 112 in a point-to-point mode where the mirrorcontrol signal for mirror 112 will vary as a function of the rangepoints to be targeted with laser pulses (and more precisely, as afunction of the order of range points to be targeted with laser pulses).

At step 608, the control circuit 106 selects a window of azimuth shotangles that are in the pool 650 at the selected elevation. The size ofthis window governs how many shot angles that the control circuit 106will order for a given batch of laser pulses 122 to be fired. Thiswindow size can be referred to as the search depth for the shotscheduling. A practitioner can configure the control circuit 106 to setthis window size based on any of a number of criteria. While the toyexamples discussed below use a window size of 3 for purposes ofillustration, it should be understood that practitioners may want to usea larger (or smaller) window size in practice. For example, in anexample embodiment, the size of the window may be a value in a rangebetween 2 shots and 12 shots. However, should the control circuit 106have larger capacities for parallel processing or should there be morelenient time constraints on latency, a practitioner may find itdesirable to choose larger window sizes. Furthermore, the controlcircuit 106 can consider a scan direction for the mirror 110 whenselecting the shot angles to include in this window. Thus, if thecontrol circuit 106 is scheduling shots for a scan directioncorresponding to increasing shot angles, the control circuit 106 canstart from the smallest shot angle in the sorted pool 650 and includeprogressively larger shot angles in the shot angle sort order of thepool 650. Similarly, if the control circuit 106 is scheduling shots fora scan direction corresponding to decreasing shot angles, the controlcircuit 106 can start from the largest shot angle in the sorted pool 650and include progressively smaller shot angles in the shot angle sortorder of the pool 650.

At step 610, the control circuit 106 determines an order for the shotangles in the selected window using the laser energy model 108 and themirror motion model 308. As discussed above, this ordering operation cancompare candidate orderings with criteria such as energy requirementsrelating to the shots to find a candidate ordering that satisfies thecriteria. Once a valid candidate ordering of shot angles is found, thiscan be used as ordered shot angles that will define the timing schedulefor the selected window of laser pulses 122. Additional details aboutexample embodiments for implementing step 610 are discussed below.

Once the shot angles in the selected window have been ordered at step610, the control circuit 106 can add these ordered shot angles to theshot list 660. As discussed in greater detail below, the shot list 660can include an ordered listing of shot angles and a scan directioncorresponding to each shot angle.

At step 612, the control circuit 106 determines whether there are anymore shot angles in pool 650 to consider at the selected elevation. Inother words, if the window size does not encompass all of the shotangles in the pool 650 at the selected elevation, then the process flowcan loop back to step 608 to grab another window of shot angles from thepool 650 for the selected elevation. If so, the process flow can thenperform steps 610 and 612 for the shot angles in this next window.

Once all of the shots have been scheduled for the shot angles at theselected elevation, the process flow can loop back from step 612 to step604 to select the next elevation from pool 650 for shot anglescheduling. As noted above, this selection can proceed in accordancewith a defined prioritization of elevations. From there, the controlcircuit 106 can perform steps 606-614 for the shot angles at the newlyselected elevation.

Meanwhile, at step 614, the control circuit 106 generates firingcommands 120 for the laser source 102 in accordance with the determinedorder of shot angles as reflected by shot list 660. By providing thesefiring commands 120 to the laser source 102, the control circuit 106triggers the laser source 102 to transmit the laser pulses 122 insynchronization with the mirrors 110 and 112 so that each laser pulse122 targets its desired range point in the field of view. Thus, if theshot list includes Shot Angles A and C to be fired at during aleft-to-right scan of the mirror 110, the control circuit 106 can usethe mirror motion model 308 to identify the times at which mirror 110will be pointing at Shot Angles A and C on a left-to-right scan andgenerate the firing commands 120 accordingly. The control circuit 106can also update the pool 650 to mark the range points corresponding tothe firing commands 120 as being “fired” to effectively remove thoserange points from the pool 650.

In the example of FIG. 6B, as noted above, the control circuit 106 seeksto schedule as many shots as possible on each scan direction of mirror110. Steps 600, 602, 604, and 606 can proceed as described above forFIG. 6A.

At step 620, the control circuit 106 selects a scan direction of mirror110 to use for scheduling. A practitioner can choose whether thisscheduling is to start with a left-to-right scan direction or aright-to-left scan direction. Then, step 608 can operate as discussedabove in connection with FIG. 6A, but where the control circuit 106 usesthe scan direction selected at step 620 to govern which shot angles areincluded in the selected window. Thus, if the selected scan directioncorresponds to increasing shot angles, the control circuit 106 can startfrom the smallest shot angle in the sorted pool 650 and includeprogressively larger shot angles in the shot angle sort order of thepool 650. Similarly, if the selected scan direction corresponds todecreasing shot angles, the control circuit 106 can start from thelargest shot angle in the sorted pool 650 and include progressivelysmaller shot angles in the shot angle sort order of the pool 650.

At step 622, the control circuit 106 determines an order for the shotangles based on the laser energy model 108 and the mirror motion model308 as discussed above for step 610, but where the control circuit 106will only schedule shot angles if the laser energy model 108 indicatesthat those shot angles are schedulable on the scan corresponding to theselected scan direction. Scheduled shot angles are added to the shotlist 660. But, if the laser energy model 108 indicates that the systemneeds to wait until the next return scan (or later) to take a shot at ashot angle in the selected window, then the scheduling of that shotangle can be deferred until the next scan direction for mirror 110 (seestep 624). This effectively returns the unscheduled shot angle to pool650 for scheduling on the next scan direction if possible.

At step 626, the control circuit 106 determines if there are any moreshot angles in pool 650 at the selected elevation that are to beconsidered for scheduling on the scan corresponding to the selected scandirection. If so, the process flow returns to step 608 to grab anotherwindow of shot angles at the selected elevation (once again taking intoconsideration the sort order of shot angles at the selected elevation inview of the selected scan direction).

Once the control circuit 106 has considered all of the shot angles atthe selected elevation for scheduling on the selected scan direction,the process flow proceeds to step 628 where a determination is made asto whether there are any more unscheduled shot angles from pool 650 atthe scheduled elevation. If so, the process flow loops back to step 620to select the next scan direction (i.e., the reverse scan direction).From there, the process flow proceeds through steps 608, 622, 624, 626,and 628 until all of the unscheduled shot angles for the selectedelevation have been scheduled and added to shot list 660. Once step 628results in a determination that all of the shot angles at the selectedelevation have been scheduled, the process flow can loop back to step604 to select the next elevation from pool 650 for shot anglescheduling. As noted above, this selection can proceed in accordancewith a defined prioritization of elevations, and the control circuit 106can perform steps 606, 620, 608, 622, 624, 626, 628, and 614 for theshot angles at the newly selected elevation.

Thus, it can be understood that the process flow of FIG. 6B will seek toschedule all of the shot angles for a given elevation during a singlescan of mirror 110 (from left-to-right or right-to-left as the case maybe) if possible in view of the laser energy model 108. However, shouldthe laser energy model 108 indicate that more time is needed to fireshots at the desired shot angles, then some of the shot angles may bescheduled for the return scan (or subsequent scan) of mirror 110.

It should also be understood that the control circuit 106 will always belistening for new range points to be targeted with new laser pulses 122.As such, steps 600 and 602 can be performed while steps 604-614 arebeing performed (for FIG. 6A) or while steps 604, 606, 620, 608, 622,624, 626, 628, and 614 are being performed (for FIG. 6B). Similarly,step 614 can be performed by the control circuit 106 while the othersteps of the FIGS. 6A and 6B process flows are being performed.Furthermore, it should be understood that the process flows of FIGS. 6Aand 6B can accommodate high priority requests for range point targeting.For example, as described in U.S. Pat. No. 10,495,757, the entiredisclosure of which is incorporated herein by reference, a request maybe received to target a set of range points in a high priority manner.Thus, the control circuit 106 can also always be listening for such highpriority requests and then cause the process flow to quickly beginscheduling the firing of laser pulses toward such range points. In acircumstance where a high priority targeting request causes the controlcircuit 106 to interrupt its previous shot scheduling, the controlcircuit 106 can effectively pause the current shot schedule, schedulethe new high priority shots (using the same scheduling techniques) andthen return to the previous shot schedule once laser pulses 122 havebeen fired at the high priority targets.

Accordingly, as the process flows of FIGS. 6A and 6B work their waythrough the list of range points in pool 650, the control circuit 106will provide improved scheduling of laser pulses 122 fired at thoserange points through use of the laser energy model 108 and mirror motionmodel 308 as compared to defined criteria such as shot energy thresholdsfor those shots. Moreover, by modeling laser energy and mirror motionover short time intervals on the order of nanoseconds using transientmodels as discussed above, these shot scheduling capabilities of thesystem can be characterized as hyper temporal because highly preciseshots with highly precise energy amounts can be accurately scheduledover short time intervals if necessary.

While FIGS. 6A and 6B show their process flows as an iterated sequenceof steps, it should be understood that if the control circuit 106 hassufficient parallelized logic resources, then many of the iterations canbe unrolled and performed in parallel without the need for return loops(or using a few number of returns through the steps). For example,different windows of shot angles at the selected elevation can beprocessed in parallel with each other if the control circuit 106 hassufficient parallelized logic capacity. Similarly, the control circuit106 can also work on scheduling for different elevations at the sametime if it has sufficient parallelized logic capacity.

FIG. 7A shows an example process flow for carrying out step 610 of FIG.6A. At step 700, the control circuit 106 creates shot angle ordercandidates from the shot angles that are within the window selected atstep 608. These candidates can be created based on the mirror motionmodel 308.

For example, as shown by FIG. 7B, the times at which the mirror 110 willtarget the different potential shot angles can be predicted using themirror motion model 308. Thus, each shot angle can be assigned a timeslot 710 with respect to the scan of mirror 110 across azimuth angles(and back). As shown by FIG. 7B, if mirror 110 starts at Angle Zero atTime 1, it will then scan to Angle A at Time 2, then scan to Angle B atTime 3, and so on through its full range of angles (which in the exampleof FIG. 7B reaches Angle J before the mirror 110 begins scanning backtoward Angle Zero). The time slots for these different angles can becomputed using the mirror motion model 308. Thus, if the window of shotangles identifies Angle A, Angle C, and Angle I as the shot angles, thenthe control circuit 106 will know which time slots of the mirror scanfor mirror 110 will target those shot angles. For example, according toFIG. 7B, Time Slots 1, 3, and 9 will target Angles A, C, and I. On thereturn scan, Time Slot 11 will also target Angle I (as shown by FIG.7B), while Time Slots 17 and 19 will also target Angles C and Arespectively. As example embodiments, the time slots 710 can correspondto time intervals in a range between around 5 nanoseconds and around 50nanoseconds, which would correspond to angular intervals of around 0.01to 0.1 degrees if mirror 110 is scanning at 12 kHz over an angularextent of 64 degrees (where +/−A is +/−16 degrees).

To create the order candidates at step 700, the control circuit 106 cangenerate different permutations of time slot sequences for differentorders of the shot angles in the selected window. Continuing with anexample where the shot angles are A, C, and I, step 700 can produce thefollowing set of example order candidates (where each order candidatecan be represented by a time slot sequence):

Order Time Slot Candidate Sequence Comments Candidate 1 1, 3, 9 Thiswould correspond to firing laser pulses in the shot angle order of ACIduring the first scan for mirror 110 (which moves from left-to-right)Candidate 2 1, 9, 17 This would correspond to firing laser pulses in theshot angle order of AIC, where laser pulses are fired at Shot Angles Aand I during the first scan for mirror 110 and where the laser pulse isfired at Shot Angle C during the second (return) scan for mirror 110(where this second scan moves from right-to-left). Candidate 3 3, 9, 19This would correspond to firing laser pulses in the shot angle order ofCIA, where laser pulses are fired at Shot Angles C and I during thefirst scan for mirror 110 and where the laser pulse is fired at ShotAngle A during the second (return) scan for mirror 110. Candidate 4 3,9, 21 This would correspond to firing laser pulses in the shot angleorder of CIA, where laser pulses are fired at Shot Angles C and I duringthe first scan for mirror 110 and where the laser pulse is fired at ShotAngle A during the third scan for mirror 110 (which moves fromleft-to-right) . . . . . . . . .

It should be understood that the control circuit 106 could createadditional candidate orderings from different permutations of time slotsequences for Shot Angles A, C, and I. A practitioner can choose tocontrol how many of such candidates will be considered by the controlcircuit 106.

At step 702, the control circuit 106 simulates the performance of thedifferent order candidates using the laser energy model 108 and thedefined shot requirements. As discussed above, these shot requirementsmay include requirements such as minimum energy thresholds for eachlaser pulse (which may be different for each shot angle), maximum energythresholds for each laser pulse (or for the laser source), and/ordesired energy levels for each laser pulse (which may be different foreach shot angle).

To reduce computational latency, this simulation and comparison withshot requirements can be performed in parallel for a plurality of thedifferent order candidates using parallelized logic resources of thecontrol circuit 106. An example of such parallelized implementation ofstep 702 is shown by FIG. 7C. In the example of FIG. 7C, steps 720, 722,and 724 are performed in parallel with respect to a plurality of thedifferent time slot sequences that serve as the order candidates. Thus,steps 720 a, 722 a, and 724 a are performed for Time Slot Sequence 1;steps 720 b, 722 b, and 724 b are performed for Time Slot Sequence 2;and so on through steps 720 n, 722 n, and 724 n for Time Slot Sequencen.

At step 720, the control circuit 106 uses the laser energy model 108 topredict the energy characteristics of the laser source and resultantlaser pulse if laser pulse shots are fired at the time slotscorresponding to the subject time slot sequence. These modeled energiescan then be compared to criteria such as a maximum laser energythreshold and a minimum laser energy threshold to determine if the timeslot sequence would be a valid sequence in view of the systemrequirements. At step 722, the control circuit 106 can label each testedtime slot sequence as valid or invalid based on this comparison betweenthe modeled energy levels and the defined energy requirements. At step724, the control circuit 106 can compute the elapsed time that would beneeded to fire all of the laser pulses for each valid time slotsequence. For example, Candidate 1 from the example above would have anelapsed time duration of 9 units of time, while Candidate 2 from theexample above would have an elapsed time duration of 17 units of time.

FIGS. 7D, 7E, and 7F show examples of such simulations of time slotsequences for our example where the shot angles to be scheduled withlaser pulses are Shot Angles A, C, and I. In this scenario, we willassume that the laser energy model 108 will employ (1) the value forE_(S) as a constant value of 1 unit of energy per unit of time and (2)the values for a and b as 0.5 each. Furthermore, we will assume thatthere are 3 units of energy left in the fiber laser 116 when the scanbegins (and where the scan begins at Angle Zero while moving fromleft-to-right). Moreover, for the purposes of this example, the energyrequirements for the shots can be defined as (8,3,4) for minimum shotenergies with respect to shot angles A, C, and I respectively, and wherethe maximum laser energy for the laser source can be defined as 20 unitsof combined seed and stored fiber energy (which would translate to amaximum laser pulse energy of 10 units of energy).

FIG. 7D shows an example result for simulating the time slot sequence oflaser pulses at time slots 1, 3, and 9. In this example, it can be seenthat this time slot sequence is invalid because the shot energy for TimeSlot 1 (targeting Shot Angle A) is only 2 units of energy, which isbelow the minimum energy threshold of 8 units for Shot Angle A. Thistime slot sequence also fails because the shot energy for Time Slot 3(targeting Shot Angle C) is only 2 units of energy, which is below theminimum energy threshold of 3 units for Shot Angle C.

FIG. 7E shows an example result for simulating the time slot sequence oflaser pulses at time slots 1, 9, and 17. In this example, it can be seenthat this time slot sequence is invalid because the shot energy for TimeSlot 1 (targeting Shot Angle A) is too low.

FIG. 7F shows an example result for simulating the time slot sequence oflaser pulses at time slots 3, 9, and 21. In this example, it can be seenthat this time slot sequence is valid because the shot energies for eachtime slot are at or above the minimum energy thresholds for theircorresponding shot angles (and none of the time slots would violate themaximum energy threshold for the laser source). It can be furthersurmised from FIG. 7F that a simulation of a Time Slot Sequence of(3,9,19) also would have failed because there is insufficient energy ina laser pulse that would have been fired at Shot Angle A.

Accordingly, the simulation of these time slot sequences would result ina determination that the time slot sequence of (3,9,21) is a validcandidate, which means that this time slot sequence can define thetiming schedule for laser pulses fired toward the shot angles in theselected window. The elapsed time for this valid candidate is 21 unitsof time.

Returning to FIG. 7A, at step 704, the control circuit 106 selects thevalid order candidate which has the lowest elapsed time. Thus, in ascenario where the simulations at step 702 would have produced two ormore valid order candidates, the control circuit 106 will select theorder candidate that will complete its firing of laser pulses thesoonest which helps improve the latency of the system.

For example embodiments, the latency with which the control circuit 106is able to determine the shot angle order and generate appropriatefiring commands is an important operational characteristic for the lidartransmitter 100. To maintain high frame rates, it is desirable for thecontrol circuit 106 to carry out the scheduling operations for all ofthe shot angles at a selected elevation in the amount of time it takesto scan mirror 110 through a full left-to-right or right-to-left scan iffeasible in view of the laser energy model 108 (where this time amountis around 40 microseconds for a 12 kHz scan frequency). Moreover, it isalso desirable for the control circuit 106 to be able to schedule shotsfor a target that is detected based on returns from shots on the currentscan line during the next return scan (e.g., when a laser pulse 122fired during the current scan detects something of interest that is tobe interrogated with additional shots (see FIG. 16 discussed above)). Inthis circumstance, the detection path for a pulse return through a lidarreceiver and into a lidar point cloud generator where the target ofinterest is detected will also need to be taken into account. Thisportion of the processing is expected to require around 0.4 to 10microseconds, which leaves around 30 microseconds for the controlcircuit 106 to schedule the new shots at the region of interest duringthe next return scan if possible. For a processor of the control circuit106 which has 2 Gflops of processing per second (which is a valueavailable from numerous FPGA and ASIC vendors), this amounts to 50operations per update, which is sufficient for the operations describedherein. For example, the control circuit 106 can maintain lookup tables(LUTs) that contain pre-computed values of shot energies for differenttime slots within the scan. Thus, the simulations of step 702 can bedriven by looking up precomputed shot energy values for the defined shotangles/time slots. The use of parallelized logic by the control circuit106 to accelerate the simulations helps contribute to the achievement ofsuch low latency. Furthermore, practitioners can adjust operationalparameters such as the window size (search depth) in a manner to achievedesired latency targets.

FIG. 8 shows an example embodiment for the lidar transmitter 100 wherethe control circuit 106 comprises a system controller 800 and a beamscanner controller 802. System controller 800 and beam scannercontroller 802 can each include a processor and memory for use incarrying out its tasks. The mirror subsystem 104 can be part of beamscanner 810 (which can also be referred to as a lidar scanner). Beamscanner controller 802 can be embedded as part of the beam scanner 810.In this example, the system controller 800 can carry out steps 600, 602,604, 608, 610, and 612 of FIG. 6A if the control circuit 106 employs theFIG. 6A process flow (or steps 600, 602, 604, 620, 608, 622, 624, 626,and 628 of FIG. 6B if the control circuit 106 employs the FIG. 6Bprocess flow), while beam scanner controller 802 carries out steps 606and 614 for the FIGS. 6A and 6B process flows. Accordingly, once thesystem controller 800 has selected the elevation and the order of shotangles, this information can be communicated from the system controller800 to the beam scanner controller 802 as shot elevation 820 and orderedshot angles 822.

The ordered shot angles 822 can also include flags that indicate thescan direction for which the shot is to be taken at each shot angle.This scan direction flag will also allow the system to recognizescenarios where the energy model indicates there is a need to pass by atime slot for a shot angle without firing a shot and then firing theshot when the scan returns to that shot angle in a subsequent time slot.For example, with reference to the example above, the scan directionflag will permit the system to distinguish between Candidate 3 (for thesequence of shot angles CIA at time slots 3, 9, and 19) versus Candidate4 (for the same sequence of shot angles CIA but at time slots 3, 9, and21). A practitioner can explicitly assign a scan direction to eachordered shot angle by adding the scan direction flag to each orderedshot angle if desired, or a practitioner indirectly assign a scandirection to each ordered shot angle by adding the scan direction flagto the ordered shot angles for which there is a change in scandirection. Together, the shot elevations 802 and order shot angles 822serve as portions of the shot list 660 used by the lidar transmitter 100to target range points with laser pulses 122.

The beam scanner controller 802 can generate control signal 806 formirror 112 based on the defined shot elevation 820 to drive mirror 112to a scan angle that targets the elevation defined by 820. Meanwhile,the control signal 804 for mirror 110 will continue to be the sinusoidalsignal that drives mirror 110 in a resonant mode. However, somepractitioners may choose to also vary control signal 804 as a functionof the ordered shot angles 822 (e.g., by varying amplitude A asdiscussed above).

In the example of FIG. 8, the mirror motion model 308 can comprise afirst mirror motion model 808 a maintained and used by the beam scannercontroller 802 and a second mirror motion model 808 b maintained andused by the system controller 800. With FIG. 8, the task of generatingthe firing commands 120 can be performed by the beam scanner controller802. The beam scanner controller 810 can include a feedback system 850that tracks the actual mirror tilt angles θ for mirror 110. Thisfeedback system 850 permits the beam scanner controller 802 to closelymonitor the actual tilt angles of mirror 110 over time which thentranslates to the actual scan angles μ of mirror 110. This knowledge canthen be used to adjust and update mirror motion model 808 a maintainedby the beam scanner controller 802. Because model 808 a will closelymatch the actual scan angles for mirror 110 due to the feedback from850, we can refer to model 808 a as the “fine” mirror motion model 808a. In this fashion, when the beam scanner controller 802 is notified ofthe ordered shot angles 822 to be targeted with laser pulses 122, thebeam scanner controller 802 can use this “fine” mirror motion model 808a to determine when the mirror has hit the time slots which target theordered shot angles 822. When these time slots are hit according to the“fine” mirror motion model 808 a, the beam scanner controller 802 cangenerate and provide corresponding firing commands 120 to the lasersource 102.

Examples of techniques that can be used for the scan tracking feedbacksystem 850 are described in the above-referenced and incorporated U.S.Pat. No. 10,078,133. For example, the feedback system 850 can employoptical feedback techniques or capacitive feedback techniques to monitorand adjust the scanning (and modeling) of mirror 110. Based oninformation from the feedback system 850, the beam scanner controller802 can determine how the actual mirror scan angles may differ from themodeled mirror scan angles in terms of frequency, phase, and/or maximumamplitude. Accordingly, the beam scanner controller 802 can thenincorporate one or more offsets or other adjustments relating thedetected errors in frequency, phase, and/or maximum amplitude into themirror motion model 808 a so that model 808 a more closely reflectsreality. This allows the beam scanner controller 802 to generate firingcommands 120 for the laser source 102 that closely match up with theactual shot angles to be targeted with the laser pulses 122.

Errors in frequency and maximum amplitude within the mirror motion model808 a can be readily derived from the tracked actual values for the tiltangle θ as the maximum amplitude A should be the maximum actual valuefor θ, and the actual frequency is measurable based on tracking the timeit takes to progress from actual values for A to −A and back.

Phased locked loops (or techniques such as PID control, both availableas software tools in MATLAB) can be used to track and adjust the phaseof the model 808 a as appropriate. The expression for the tilt angle θthat includes a phase component (p) can be given as:

θ=A cos(2πft+p)

From this, we can recover the value for the phase p by the relation:

θ≈A cos(2πft)−A sin(2πft)p

Solving for p, this yields the expression:

$p = \frac{{{Acos}\left( {2\pi{ft}} \right)} - \theta}{{Asin}\left( {2\pi{ft}} \right)}$

Given that the tracked values for A, f, t, and θ are each known, thevalue for p can be readily computed. It should be understood that thisexpression for p assumes that the value of the p is small, which will bean accurate assumption if the actual values for A, f, t, and θ areupdated frequently and the phase is also updated frequently. Thiscomputed value of p can then be used by the “fine” mirror motion model808 a to closely track the actual shot angles for mirror 110, andidentify the time slots that correspond to those shot angles accordingto the expression:

$t = \frac{{\arccos\left( \frac{\mu - \varphi}{2A} \right)} - p}{2\pi f}$

While a practitioner will find it desirable for the beam scannercontroller 802 to rely on the highly accurate “fine” mirror motion model808 a when deciding when the firing commands 120 are to be generated,the practitioner may also find that the shot scheduling operations cansuffice with less accurate mirror motion modeling. Accordingly, thesystem controller 800 can maintain its own model 808 b, and this model808 b can be less accurate than model 808 a as small inaccuracies in themodel 808 b will not materially affect the energy modeling used todecide on the ordered shot angles 822. In this regard, model 808 b canbe referred to as a “coarse” mirror motion model 808 b. If desired, apractitioner can further communicate feedback from the beam scannercontroller 802 to the system controller 800 so the system controller 800can also adjusts its model 808 b to reflect the updates made to model808 a. In such a circumstance, the practitioner can also decide on howfrequently the system will pass these updates from model 808 a to model808 b.

Marker Shots to Bleed Off and/or Regulate Shot Energy:

FIG. 9 depicts an example process flow for execution by the controlcircuit 106 to insert marker shots into the shot list in order to bleedoff energy from the laser source 102 when needed. As discussed above,the control circuit 106 can consult the laser energy model 108 asapplied to the range points to be targeted with laser pulses 122 todetermine whether a laser energy threshold would be violated. If so, thecontrol circuit 106 may insert a marker shot into the shot list to bleedenergy out of the laser source 102 (step 902). In an example embodiment,this threshold can be set to define a maximum or peak laser energythreshold so as to avoid damage to the laser source 102. In anotherexample embodiment, this threshold can be set to achieve a desiredconsistency, smoothness, and/or balance in the energies of the laserpulse shots.

For example, one or more marker shots can be fired to bleed off energyso that a later targeted laser pulse shot (or set of targeted shots)exhibits a desired amount of energy. As an example embodiment, themarker shots can be used to bleed off energy so that the targeted laserpulse shots exhibit consistent energy levels despite a variable rate offiring for the targeted laser pulse shots (e.g., so that the targetedlaser pulse shots will exhibit X units of energy (plus or minus sometolerance) even if those targeted laser pulse shots are irregularlyspaced in time). The control circuit 106 can consult the laser energymodel 108 to determine when such marker shots should be fired toregulate the targeted laser pulse shots in this manner.

Modeling Eye and Camera Safety Over Time:

FIG. 10 depicts an example process flow for execution by the controlcircuit 106 where eye safety requirements are also used to define oradjust the shot list. To support these operations, the control circuit106 can also, at step 1000, maintain an eye safety model 1002. Eyesafety requirements for a lidar transmitter 100 may be established todefine a maximum amount of energy that can be delivered within a definedspatial area in the field of view over a defined time period. Since thesystem is able to model per pulse laser energy with respect to preciselytargeted range points over highly granular time periods, this allows thecontrol circuit 106 to also monitor whether a shot list portion wouldviolate eye safety requirements. Thus, the eye safety model 1002 canmodel how much aggregated laser energy is delivered to the definedspatial area over the defined time period based on the modeling producedfrom the laser energy model 108 and the mirror motion model 308. At step1010, the control circuit 106 uses the eye safety model 1002 todetermine whether the modeled laser energy that would result from asimulated sequence of shots would violate the eye safety requirements.If so, the control circuit can adjust the shot list to comply with theeye safety requirements (e.g., by inserting longer delays betweenordered shots delivered close in space, by re-ordering the shots, etc.)

FIG. 11 shows an example lidar transmitter 100 that is similar in natureto the example of FIG. 8, but where the system controller 800 alsoconsiders the eye safety model 1002 when deciding on how to order theshot angles. FIG. 12 shows how the simulation step 702 from FIG. 7A canbe performed in example embodiments where the eye safety model 1002 isused. As shown by FIG. 12, each parallel path can include steps 720,722, and 724 as discussed above. Each parallel path can also include astep 1200 to be performed prior to step 722 where the control circuit106 uses the eye safety model 1002 to test whether the modeled laserenergy for the subject time slot sequence would violate eye safetyrequirements. If the subject time slot sequence complies with thecriteria tested at steps 720 and 1200, then the subject time slotsequence can be labeled as valid. If the subject time slot sequenceviolates the criteria tested at steps 720 or 1200, then the subject timeslot sequence can be labeled as invalid.

Similar to the techniques described for eye safety in connection withFIGS. 10, 11, and 12, it should be understood that a practitioner canalso use the control circuit to model and evaluate whether time slotsequences would violate defined camera safety requirements. To reducethe risk of laser pulses 122 impacting on and damaging cameras in thefield of view, the control circuit can also employ a camera safety modelin a similar manner and toward similar ends as the eye safety model1002. In the camera safety scenario, the control circuit 106 can respondto detections of objects classified as cameras in the field of view bymonitoring how much aggregated laser energy will impact that cameraobject over time. If the model indicates that the camera object wouldhave too much laser energy incident on it in too short of a time period,the control circuit can adjust the shot list as appropriate.

Moreover, as noted above with respect to the laser energy model 108 andthe mirror motion model 308, the eye safety and camera safety models cantrack aggregated energy delivered to defined spatial areas over definedtime periods over short time intervals, and such short interval eyesafety and camera safety models can be referred to as transient eyesafety and camera safety models.

Additional Example Embodiments

FIG. 13 shows another example of a process flow for the control circuit106 with respect to using the models to dynamically determine the shotlist for the transmitter 100.

At step 1300, the laser energy model 108 and mirror motion model 308 areestablished. This can include determining from factory or calibrationthe values to be used in the models for parameters such as E_(P), a, b,and A. Step 1300 can also include establishing the eye safety model 1002by defining values for parameters that govern such a model (e.g.parameters indicative of limits for aggregated energy for a definedspatial area over a defined time period). At step 1302, the control lawfor the system is connected to the models established at step 1300.

At step 1304, the seed energy model used by the laser energy model 108is adjusted to account for nonlinearities. This can employ the clipped,offset (affine) model for seed energy as discussed above.

At step 1306, the laser energy model 108 can be updated based on lidarreturn data and other feedback from the system. For example, as notedabove in connection with FIG. 2D, the actual energies in laser pulses122 can be derived from the pulse return data included in point cloud256. For example, the pulse return energy can be modeled as a functionof the transmitted pulse energy according to the following expression(for returns from objects that are equal to or exceed the laser spotsize and assuming modest atmospheric attenuation):

${{Pulse}{Return}{Energy}} = {\left( \frac{{PEAperture}_{Receiver}}{\pi R^{2}} \right){Reflectivity}}$

In this expression, Pulse Return Energy represents the energy of thepulse return (which is known from the point cloud 256), PE representsthe unknown energy of the transmitted laser pulse 122,Aperture_(Receiver) represents the known aperture of the lidar receiver(see 1400 in FIG. 14), R represents the measured range for the return(which is known from the point cloud 256), and Reflectivity representsthe percentage of reflectivity for the object from which the return wasreceived. Therefore, one can solve for PE so long as the reflectivity isknown. This will be the case for objects like road signs whosereflectivity is governed by regulatory agencies. Accordingly, by usingreturns from known fiducials such as road signs, the control circuit 106can derive the actual energy of the transmitted laser pulse 122 and usethis value to facilitate determinations as to whether any adjustments tothe laser energy model 108 are needed (e.g., see discussions above reupdating the values for a and b based on PE values which represent theactual energies of the transmitted laser pulses 122).

Also, at step 1308, the laser health can be assessed and monitored as abackground task. The information derived from the feedback received forsteps 1306 and 1308 can be used to update model parameters as discussedabove. For example, as noted above, the values for the seed energy modelparameters as well as the values for a and b can be updated by measuringthe energy produced by the laser source 102 and fitting the data to theparameters. Techniques which can be used for this process include leastsquares, sample matrix inversion, regression, and multiple exponentialextensions. Further still, as noted above, the amount of error can bereduced by using known targets with a given reflectivity and using theseto calibrate the system. This is helpful because the reflectivity of aquantity that is known, i.e. a fiducial, allows one to explicitlyextract shot energy (after backing out range dependencies and anyobliquity). Examples of fiducials that may be employed include roadsigns and license plates.

At step 1310, the lidar return data and the coupled models can be usedto ensure that the laser pulse energy does not exceed safety levels.These safety levels can include eye safety as well as camera safety asdiscussed above. Without step 1310, the system may need to employ a muchmore stringent energy requirement using trial and error to establishlaser settings to ensure safety. For example if we only had a lasermodel where the shot energy is accurate to only ±3 J per shot around thepredicted shot, and maximum shot energy is limited to 8, we could notuse any shots predicted to exceed 5. However, the hyper temporalmodeling and control that is available from the laser energy model 108and mirror motion model 308 as discussed herein allows us to obtainaccurate predictions within a few percent error, virtually erasing theoperational lidar impact of margin.

At step 1312, the coupled models are used with different orderings ofshots, thereby obtaining a predicted shot energy in any chosen orderedsequence of shots drawn from the specified list of range points. Step1312 may employ simulations to predict shot energies for different timeslots of shots as discussed above.

At step 1314, the system inserts marker shots in the timing schedule ifthe models predict that too much energy will build up in the lasersource 102 for a given shot sequence. This reduces the risk of too muchenergy being transferred into the fiber laser 116 and causing damage tothe fiber laser 116.

At step 1316, the system determines the shot energy that is needed todetect targets with each shot. These values can be specified as aminimum energy threshold for each shot. The value for such threshold(s)can be determined from radiometric modeling of the lidar, and theassumed range and reflectivity of a candidate target. In general, thisstep can be a combination of modeling assumptions as well asmeasurements. For example, we may have already detected a target, so thesystem may already know the range (within some tolerance). Since theenergy required for detection is expected to vary as the square of therange, this knowledge would permit the system to establish the minimumpulse energy thresholds so that there will be sufficient energy in theshots to detect the targets.

Steps 1318 and 1320 operate to prune the candidate ordering optionsbased on the energy requirements (e.g., minimum energy thresholds pershot) (for step 1318) and shot list firing completion times (to favorvalid candidate orderings with faster completion times) (for step 1320).

At step 1322, candidate orderings are formed using elevation movementson both scan directions. This allows the system to consider taking shotson both a left-to-right scan and a right-to-left scan. For example,suppose that the range point list has been completed on a certainelevation, when the mirror is close to the left hand side. Then it isfaster to move the elevation mirror at that point in time and begin thefresh window of range points to be scheduled beginning on this same lefthand side and moving right. Conversely, if we deplete the range pointlist when the mirror is closer to the right hand side it is faster tomove the mirror in elevation whilst it is on the right hand side.Moreover, in choosing an order from among the order candidates, and whenmoving from one elevation to another, movement on either side of themirror motion, the system may move to a new elevation when mirror 110 isat one of its scan extremes (full left or full right). However, ininstances where a benefit may arise from changing elevations when mirror110 is not at one of its scan extremes, the system may implementinterline skipping as described in the above-referenced and incorporatedU.S. Pat. No. 10,078,133. The mirror motion model 308 can also beadjusted to accommodate potential elevation shift during a horizontalscan.

At step 1324, if processing time allows the control circuit 106 toimplement auctioning (whereby multiple order candidates areinvestigated, the lowest “cost” (e.g., fastest lidar execution time)order candidate is selected by the control circuit 106 (acting as“auctioneer”). A practitioner may not want the control circuit toconsider all of the possible order candidates as this may be toocomputationally expensive and introduce an undue amount of latency.Thus, the control circuit 106 can enforce maximums or other controls onhow many order candidates are considered per batch of shots to beordered. Greedy algorithms can be used when choosing ordering shots.Generally, the system can use a search depth value (which defines howmany shots ahead the control circuit will evaluate) in this process in amanner that is consistent with any real time consideration in shot listgeneration. At step 1326, delays can be added in the shot sequence tosuppress a set of shots and thus increase available shot energy toenable a finer (denser) grid as discussed above. The methodology forsorting through different order candidates can be considered a specialcase of the Viterbi algorithm which can be implemented using availablesoftware packages such as Mathworks. This can also be inferred usingequivalence classes or group theoretic methods. Furthermore, if thesystem detects that reduced latency is needed, the search depth can bereduced (see step 1328).

FIG. 14 depicts an example embodiment for a lidar transmitter 100 thatshows how the system controller 800 can interact with the lidar receiver1400 to coordinate system operations. The lidar receiver 1400 canreceive and process pulse returns 1402 to compute range information forobjects in the field of view impacted by the laser pulses 122. Thisrange information can then be included in the point cloud 1404 generatedby the lidar system. Examples of suitable technology for use as thelidar receiver 1400 are described in U.S. Pat. Nos. 9,933,513 and10,754,015, the entire disclosures of which are incorporated herein byreference. In the example of FIG. 14, the system controller 800 can usethe point cloud 1404 to intelligently select range points for targetingwith laser pulses, as discussed in the above-referenced and incorporatedpatents. For example, the point cloud data can be used to determineranges for objects in the field of view that are to be targeted withlaser pulses 122. The control circuit 106 can use this range informationto determine desired energy levels for the laser pulses 122 which willtarget range points that are believed to correspond to those objects. Inthis fashion, the control circuit 106 can controllably adjust the laserpulse energy as a function of the estimated range of the object beingtargeted so the object is illuminated with a sufficient amount of lightenergy given its estimated range to facilitate adequate detection by thelidar receiver 1400. Further still, the beam scanner controller 802 canprovide shot timing information 1410 to the receiver 1400 and the systemcontroller 800 can provide shot data 1412 (such as data identifying thetargeting range points) to the receiver 1400. The combination of thisinformation informs the receiver how to control which pixels of thereceiver 1400 should be activated for detecting pulse returns 1402(including when those pixels should be activated). As discussed in theabove-referenced and incorporated '513 and '015 patents, the receivercan select pixels for activation to detect pulse returns 1402 based onthe locations of the targeted range points in the field of view.Accordingly, precise knowledge of which range points were targeted andwhen those range points were targeted helps improve the operations ofreceiver 1400. Although not shown in FIG. 14, it should also beunderstood that a practitioner may choose to also include a camera thatimages the field of view, and this camera can be optically co-axial(co-bore sighted) with the lidar transmitter 100. Camera images can alsobe used to facilitate intelligent range point selection among othertasks.

FIG. 15 shows another example of a process flow for the control circuit106 with respect to using the models to dynamically determine the shotlist for the transmitter 100. At step 1500, the laser energy model 108and mirror motion model 308 are established. This can operate like step1300 discussed above. At step 1502, the model parameters are updatedusing pulse return statistics (which may be derived from point cloud1404 or other information provided by the receiver 1400) and mirror scanposition feedback (e.g., from feedback system 850). At step 1504, themodels are coupled so that shot angles are assigned to time slotsaccording to the mirror motion model 308 for which shot energies can bepredicted according to the laser energy model 108. These coupled modelscan then be embedded in the shot scheduling logic used by controlcircuit 106. At step 1506, a list of range points to be targeted withlaser pulses 122 is received. At step 1508, a selection is made for thesearch depth that governs how far ahead the system will schedule shots.

Based on the listed range points and the defined search depth, the ordercandidates for laser pulse shots are created (step 1510). The mirrormotion model 308 can assign time slots to these order candidates asdiscussed above. At step 1512, each candidate is tested using the laserenergy model 108. This testing may also include testing based on the eyesafety model 1002 and a camera safety model. This testing can evaluatethe order candidates for compliance with criteria such as peak energyconstraints, eye safety constraints, camera safety constraints, minimumenergy thresholds, and completion times. If a valid order candidate isfound, the system can fire laser pulses in accordance with thetiming/sequencing defined by the fastest of the valid order candidates.Otherwise, the process flow can return to step 1510 to continue thesearch for a valid order candidate.

Controllable Detection Intervals for Return Processing:

In accordance with another example embodiment, the shot list can be usedto exercise control over how the lidar receiver 1400 detects returnsfrom laser pulse shots 122. FIG. 18A shows an example lidar receiver1400 for use in a lidar system. The lidar receiver 1400 comprisesphotodetector circuitry 1800 which includes a photodetector array 1802.The photodetector array 1802 comprises a plurality of detector pixels1804 that sense incident light and produce a signal representative ofthe sensed incident light. The detector pixels 1804 can be organized inthe photodetector array 1802 in any of a number of patterns. In someexample embodiments, the photodetector array 1802 can be atwo-dimensional (2D) array of detector pixels 1804. However, it shouldbe understood that other example embodiments may employ aone-dimensional (1D) array of detector pixels 1804 if desired by apractitioner.

The photodetector circuitry 1800 generates a return signal 1806 inresponse to a pulse return 1402 that is incident on the photodetectorarray 1802. The choice of which detector pixels 1804 to use forcollecting a return signal 1806 corresponding to a given return 1402 canbe made based on where the laser pulse shot 122 corresponding to thereturn 1402 that was targeted. Thus, if a laser pulse shot 122 istargeting a range point located as a particular azimuth angle, elevationangle pair; then the lidar receiver can map that azimuth, elevationangle pair to a set of pixels 1804 within the array 1802 that will beused to detect the return 1402 from that laser pulse shot 122. Themapped pixel set can include one or more of the detector pixels 1804.This pixel set can then be activated and read out from to supportdetection of the subject return 1402 (while the pixels outside the pixelset are deactivated so as to minimize potential obscuration of thereturn 1402 within the return signal 1806 by ambient or interferinglight that is not part of the return 1402 but would be part of thereturn signal 1806 if unnecessary pixels 1804 were activated when return1402 was incident on array 1802). In this fashion, the lidar receiver1400 will select different pixel sets of the array 1802 for readout in asequenced pattern that follows the sequenced spatial pattern of thelaser pulse shots 122. Return signals 1806 can be read out from theselected pixel sets, and these return signals 1806 can be processed todetect returns 1402 therewithin.

FIG. 18A shows an example where one of the pixels 1804 is turned on tostart collection of a sensed signal that represents incident light onthat pixel (to support detection of a return 1402 within the collectedsignal), while the other pixels 1804 are turned off (or at least notselected for readout). While the example of FIG. 18A shows a singlepixel 1804 being included in the pixel set selected for readout, itshould be understood that a practitioner may prefer that multiple pixels1804 be included in one or more of the selected pixel sets. For example,it may be desirable to include in the selected pixel set one or morepixels 1804 that are adjacent to the pixel 1804 where the return 1402 isexpected to strike.

Examples of circuitry and control logic that can used for this selectivepixel set readout are described in U.S. Pat. Nos. 10,754,015 and10,641,873, the entire disclosures of each of which are incorporatedherein by reference. These incorporated patents also describe exampleembodiments for the photodetector circuitry 1800, including the use of amultiplexer to selectively read out signals from desired pixel sets aswell as an amplifier stage positioned between the photodetector array1802 and multiplexer.

Signal processing circuit 1820 operates on the return signal 1806 tocompute return information 1822 for the targeted range points, where thereturn information 1822 is added to the lidar point cloud 1404. Thereturn information 1822 may include, for example, data that represents arange to the targeted range point, an intensity corresponding to thetargeted range point, an angle to the targeted range point, etc. Asdescribed in the above-referenced and incorporated '015 and '873patents, the signal processing circuit 1820 can include ananalog-to-digital converter (ADC) that converts the return signal 1806into a plurality of digital samples. The signal processing circuit 1820can process these digital samples to detect the returns 1402 and computethe return information 1822 corresponding to the returns 1402. In anexample embodiment, the signal processing circuit 1820 can perform timeof flight (TOF) measurement to compute range information for the returns1402. However, if desired by a practitioner, the signal processingcircuit 1820 could employ time-to-digital conversion (TDC) to computethe range information. Additional details about how the signalprocessing circuit 1820 can operate for an example embodiment arediscussed below.

The lidar receiver 1400 can also include circuitry that can serve aspart of the control circuit 106 of the lidar system. This controlcircuitry is shown as a receiver controller 1810 in FIG. 18A. Thereceiver controller 1810 can process scheduled shot information 1812 togenerate the control data 1814 that defines which pixel set to select(and when to use each pixel set) for detecting returns 1402. Thescheduled shot information 1812 can include shot data information thatidentifies timing and target coordinates for the laser pulse shots 122to be fired by the lidar transmitter 100. In an example embodiment, thescheduled shot information 1812 can also include detection range valuesto use for each scheduled shot to support the detection of returns 1412from those scheduled shots. These detection range values may includeminimum and maximum range values (Rmin and Rmax respectively) for eachshot. In this fashion, Rmin(i) would be the minimum detection rangeassociated with Shot(i) and Rmax(i) would be the maximum detection rangeassociated with Shot(i). These minimum and maximum range values can betranslated by the receiver controller 1810 into times for starting andstopping collections from the selected pixels 1804 of the array 1802 asdiscussed below.

The receiver controller 1810 and/or signal processing circuit 1820 mayinclude one or more processors. These one or more processors may takeany of a number of forms. For example, the processor(s) may comprise oneor more microprocessors. The processor(s) may also comprise one or moremulti-core processors. As another example, the one or more processorscan take the form of a field programmable gate array (FPGA) orapplication-specific integrated circuit (ASIC) which provideparallelized hardware logic for implementing their respectiveoperations. The FPGA and/or ASIC (or other compute resource(s)) can beincluded as part of a system on a chip (SoC). However, it should beunderstood that other architectures for such processor(s) could be used,including software-based decision-making and/or hybrid architectureswhich employ both software-based and hardware-based decision-making. Theprocessing logic implemented by the receiver controller 1810 and/orsignal processing circuit 1820 can be defined by machine-readable codethat is resident on a non-transitory machine-readable storage mediumsuch as memory within or available to the receiver controller 1810and/or signal processing circuit 1820. The code can take the form ofsoftware or firmware that define the processing operations discussedherein. This code can be downloaded onto the processor using any of anumber of techniques, such as a direct download via a wired connectionas well as over-the-air downloads via wireless networks, which mayinclude secured wireless networks. As such, it should be understood thatthe lidar receiver 1400 can also include a network interface that isconfigured to receive such over-the-air downloads and update theprocessor(s) with new software and/or firmware. This can be particularlyadvantageous for adjusting the lidar receiver 1400 to changingregulatory environments. When using code provisioned for over-the-airupdates, the lidar receiver 1400 can operate with unidirectionalmessaging to retain function safety.

FIGS. 19A and 19B illustrate time constraints involved in detecting shotreturns 1402 and how these time constraints relate to the Rmin and Rmaxvalues.

In FIGS. 19A and 19B, the horizontal axes correspond to time. In FIG.19A, the time at which a first laser pulse shot (Shot 1) is fired isdenoted by T(1) (where the parenthetical (1) in T(1) references the shotnumber). TT1(1) denotes when the receiver 1400 starts the collectionfrom the pixel set used for detecting a return from Shot 1. The time atwhich collection stops from this pixel set (to end the readout of signalfrom the pixel set for detecting a return from Shot 1) is denoted asTT2(2). It should be understood that the parentheticals in these termsT, TT1, and TT2 reference the shot number for the detection. Thus, thetime duration from TT1(1) to TT2(1) represents the detection intervalfor detecting a return from Shot 1 because it is during this timeinterval that the receiver 1400 is able to collect signal from the pixelset used for detecting a return from Shot 1.

As shown by FIG. 19A, the time duration from T(1) to TT1(1) correspondsto the minimum range that must exist to the target (relative to thereceiver 1400) in order to detect a return from Shot 1. This minimumrange is denoted by Rmin(1) in FIG. 19A (where the parentheticalreferences the shot number to which the minimum range value isapplicable). If the target were located less than Rmin(1) from thereceiver 1400, then the receiver 1400 would not be able to detect thereturn from Shot 1 because the lidar receiver would not yet have startedcollection from the pixel set used for detecting that return.

FIG. 19A also shows that the time duration from T(1) to TT2(1)corresponds to the maximum range to the target for detecting a returnfrom Shot 1. This maximum range is denoted by Rmax(1) in FIG. 19A (wherethe parenthetical references the shot number to which the maximum rangevalue is applicable). If the target were located greater than Rmax(1)from the receiver 1400, then the receiver 1400 would not be able todetect the return from Shot 1 because the lidar receiver would havealready stopped collection from the pixel set used for detecting thatreturn by the time that the return strikes that pixel set.

Thus, so long as the target for Shot 1 is located at a range betweenRmin(1) and Rmax(1), the receiver 1400 is expected to be capable ofdetecting the return if collection from the pixel set starts at timeTT1(1) and stops at time TT2(1). The range interval encompassed by thedetection interval of TT1(1) to TT2(1) can be referred to as the rangeswath S(1) (where the parenthetical references the shot number to whichthe range swath is applicable). This range swath can also be referencedas a range buffer as it represents a buffer of ranges for the targetthat make the target detectable by the receiver 1400.

FIG. 19B further extends these time-range relationships by adding asecond shot (Shot 2). The time at which Shot 2 is fired by the lidartransmitter 100 is denoted as T(2) in FIG. 19B. The start collectiontime for the pixel set used to detect the return from Shot 2 is denotedas TT1(2) in FIG. 19B, and the stop collection time for the pixel setused with respect to detecting the return from Shot 2 is denoted asTT2(2) in FIG. 19B. FIG. 19B further shows that (1) the time durationfrom T(2) to TT1(2) corresponds to the minimum range to the target fordetecting a return from Shot 2 and (2) the time duration from T(2) toTT2(2) corresponds to the maximum range to the target for detecting areturn from Shot 2. These minimum and maximum range values are denotedas Rmin(2) and Rmax(2) respectively by FIG. 19B. The range swath S(2)defines the range interval (or range buffer) between Rmin(2) and Rmax(2)for the detection interval of TT1(2) to TT2(2).

In an example embodiment, the photodetector circuitry 1800 is capable ofsensing returns from one pixel set at a time. For such an exampleembodiment, the detection interval for detecting a return for a givenshot cannot overlap with the detection interval for detecting a returnfrom another shot. This means that TT1(2) should be greater than orequal to TT2(1), which then serves as a constraint on the choice ofstart and stop collection times for the pixel clusters. However, itshould be understood that this constraint could be eliminated with otherexample embodiments through the use of multiple readout channels for thelidar receiver 1400 as discussed below in connection with FIG. 26.

As noted above, each detection interval (D(i), which corresponds to(TT1(i) to TT2(i)) will be associated with a particular laser pulse shot(Shot(i)). The system can control these shot-specific detectionintervals so that they can vary across different shots. As such, thedetection interval of D(j) for Shot(j) can have a different durationthan the detection interval of D(k) for Shot(k).

Moreover, as noted above, each detection interval D(i) has acorresponding shot interval SI(i), where the shot interval SI(i)corresponding to D(i) can be represented by the interval from shot timeT(i) to the shot time T(i+1). Thus, consider a shot sequence of Shots1-4 at times T(1), T(2), T(3), and T(4) respectively. For this shotsequence, detection interval D(1) for detecting the return from Shot(1)would have a corresponding shot interval SI(1) represented by the timeinterval from T(1) to T(2). Similarly, detection interval D(2) fordetecting the return from Shot(2) would have a corresponding shotinterval SI(2) represented by the time interval from T(2) to T(3); andthe detection interval D(3) for detecting the return from Shot(3) wouldhave a corresponding shot interval SI(3) represented by the timeinterval from T(3) to T(4). Counterintuitively, the inventors have foundthat it is often not desirable for a detection interval to be of thesame duration as its corresponding shot interval due to factors such asthe amount of processing time that is needed to detect returns withinreturn signals (as discussed in greater detail below). In many cases, itwill be desirable for the control process to define a detection intervalso that it exhibits a duration shorter than the duration of itscorresponding shot interval (D(i)<SI(i)). In this fashion, processingresources in the signal processing circuit 1820 can be better utilized,as discussed below. Furthermore, in some other cases, it may bedesirable for the variability of the detection intervals relative totheir corresponding shot intervals to operate where a detection intervalexhibits a duration longer than the duration of its corresponding shotinterval (D(i)>SI(i)). For example, if the next shot at T(i+1) has anassociated Rmin value greater than zero, and where the shot at T(i) istargeting a range point expected to be at a long range while the shot atT(i+1) is targeting a range point expected to be at medium or longrange, then it may be desirable for D(i) to be greater than SI(i).

It can be appreciated that a laser pulse shot, Shot(i), fired at timeT(i) will be traveling at the speed of light. On this basis, and usingthe minimum and maximum range values of Rmin(i) and Rmax(i) fordetecting the return from Shot(i), the minimum roundtrip distance forShot(i) and its return would be 2Rmin(i) and the minimum roundtrip timefor Shot(i) and its return would be TT1(i)−T(i). The value for TT1(i)could be derived from Rmin(i) according to these relationships asfollows (where the term c represents the speed of light):

(TT1(i)−T(i))c=2Rmin(i)

-   -   which can be re-expressed as:

${{TT}1(i)} = {\frac{2{{Rmin}(i)}}{c} + {T(i)}}$

Thus, knowledge of when Shot(i) is fired and knowledge of the value forRmin(i) allows the receiver 1400 to define when collection should startfrom the pixel set to be used for detecting the return from Shot (i).

Similarly, the value for TT2(i) can be derived from Rmax(i) according tothese relationships as follows (where the term c represents the speed oflight):

(TT2(i)−T(i))c=2Rmax(i)

-   -   which can be re-expressed as:

${{TT}2(i)} = {\frac{2{{Rmax}(i)}}{c} + {T(i)}}$

Thus, knowledge of when Shot(i) is fired and knowledge of the value forRmax(i) allows the receiver 1400 to define when collection can stop fromthe pixel set to be used for detecting the return from Shot(i).

A control process for the lidar system can then operate to determinesuitable Rmin(i) and Rmax(i) values for detecting the returns from eachShot(i). These Rmin, Rmax pairs can then be translated into appropriatestart and stop collection times (the on/off times of TT1 and TT2) foreach shot. In an example embodiment, if the lidar point cloud 1404 hasrange data and location data about a plurality of objects of interest ina field of view for the receiver 1400, this range data and location datacan be used to define current range estimates for the objects ofinterest, and suitable Rmin, Rmax values for detecting returns fromlaser pulse shots that target range points corresponding to where theseobjects of interest are located can be derived from these rangeestimates. In another example embodiment, the control process for thelidar system can access map data based on the geographic location of thereceiver 1400. From this map data, the control process can deriveinformation about the environment of the receiver 1400, and suitableRmin, Rmax values can be derived from this environmental information.Additional example embodiments for determine the values for the Rmin,Rmax pairs are discussed below.

FIG. 18B depicts an example process flow for execution by the receivercontroller 1810 for the lidar receiver 1400 of FIG. 18A to control theselection of detector pixels 1804 in the photodetector array 1802 forreadout. In this example, the lidar system includes a buffer 1830 inwhich the scheduled shot information 1812 is buffered by the controlcircuit 106. This scheduled shot information 1812 can include, for eachscheduled laser pulse shot, data that identifies the time the shot is tobe fired, data that identifies range point to be targeted with thesubject shot (e.g., the azimuth and elevation angles at which thesubject shot will be fired), and data that defines the detectioninterval for detecting the return from the subject shot (e.g., data thatidentifies the Rmin and Rmax values to use for detecting the return fromthe subject shot). The entries 1832 in buffer 1830 thus correspond tothe shot time, azimuth angle, elevation angle, Rmin, and Rmax values foreach shot. Moreover, the order in which these entries are stored in thebuffer 1832 can define the shot schedule. For example, the buffer can bea first in/first out (FIFO) buffer where entries 1832 are added into andread out of the buffer in accordance with the order in which the shotsare to be fired.

Steps 1850, 1852, and 1854 of FIG. 18B define a translation process flowfor the receiver controller 1810. At step 1850, the receiver controller1810 reads the entry 1832 corresponding to the first shot to determinethe shot time, shot angles, and range swath information for that shot.At step 1852, the receiver controller 1852 determines which pixel set ofthe photodetector array 1800 to select for detecting the return from theshot defined by the shot angles of the entry 1832 read at step 1850.This can be accomplished by mapping the azimuth and elevation angles forthe shot to a particular pixel set of the array 1802. Thus, step 1852can operate to generate data that identifies one or more pixels 1804 ofthe array 1802 to include in the pixel set for detecting the return fromthe subject shot. The above-referenced and incorporated '015 and '873patents describe how a practitioner can implement this mapping of shotlocations to pixels. At step 1854, the receiver controller 1810translates the shot time and Rmin, Rmax values into the TT1, TT2 values.This translation can use the expressions discussed above for computingTT1(i) and TT2(i) as a function of T(i), Rmin(i), and Rmax(i) for agiven Shot(i). The values for the determined pixel set and thestart/stop collection times for the pixel set can then be added tobuffer 1840 as a new entry 1842. The process flow can then return tostep 1850 to iterate through steps 1850-1854 and generate the controldata for the next shot (as defined by the next entry 1832 in buffer1830), and so on for so long as there are new entries 1832 in buffer1830.

Steps 1856, 1858, and 1860 of FIG. 18B define a readout control processflow for the receiver controller 1810. Each entry 1842 in buffer 1840corresponds to exercising control over detecting the return from adifferent shot and identifies the pixel set to use for the detection aswell as the start and stop collection times for that detection (TT1, TT2values). At step 1856, the receiver controller reads the entry 1842corresponding to the first shot to identify the pixel set, TT1, and TT2values for detecting the return from that shot. Then, at the timecorresponding to TT1, the receiver controller 1810 starts collecting thesensed signal from the identified pixel set (step 1860). In thisfashion, the receiver controller 1810 can provide control data 1814 tothe photodetector circuitry 1800 at the time corresponding to TT1 thatinstructs the photodetector circuitry 1800 to start the signal readoutfrom the pixel(s) within the identified pixel set. Next, at the timecorresponding to TT2, the receiver controller 1810 stops the collectionfrom the identified pixel set (step 1862). To accomplish this, thereceiver controller 1810 can provide control data 1814 to thephotodetector circuitry 1800 at the time corresponding to TT2 thatinstructs the photodetector circuitry 1800 to stop the readout from thepixel(s) within the identified pixel set. From there, the process flowreturns to step 1856 to continue the readout control process flow forthe next entry 1842 in buffer 1840. This iteration through steps1856-1860 continues for so long as there are new entries 1842 in buffer1840 to process.

As discussed above in connection with FIGS. 8, 11, and 14, apractitioner may want the lidar system to exercise highly precisecontrol over when the laser pulse shots are fired; and this can beaccomplished by having the beam scanner controller 802 provide firingcommands 120 to the laser source 102 precisely when the fine mirrormotion model 808 a indicates the lidar transmitter 100 will be pointingat a particular shot angle (e.g., an azimuth angle, elevation anglepair). The beam scanner controller 802 can then report these preciseshot times to the receiver 1400 as shot timing information 1412.Accordingly, as shown by FIG. 18C, the lidar receiver 1400 can receivethe scheduled shot information 1812 in the form of (1) the shot timinginformation 1410 from the beam scanner controller 802 (which in thisexample will identify the precise shot times for each shot) and (2) theshot data 1312 from the system controller 800 (which in this examplewill identify the shot angles for each shot as well as the Rmin and Rmaxvalues for each shot).

FIG. 18D depicts an example process flow for the receiver controller1810 with respect to the example of FIG. 18C. The process flow of FIG.18D can operate in a similar fashion as the process flow of FIG. 18Dwith a couple of exceptions. For example, at step 1854 of FIG. 18D, thereceiver controller 1810 can compute the start and stop collection timesas time offsets relative to the fire time for the shot rather than asabsolute values. That is, rather than computing values for TT1(i) andTT2(i), the receiver controller can compute values for (1) TT1(i)−T(i)(which would identify the TT1(i) offset relative to fire time T(i)) and(2) TT2(i)−T(i) (which would identify the TT2(i) offset relative to firetime T(i)) as follows:

${{{TT}1(i){Offset}} = {{{{TT}1(i)} - {T(i)}} = \frac{2{{Rmin}(i)}}{c}}}{{{TT}2(i){Offset}} = {{{{TT}2(i)} - {T(i)}} = \frac{2{{Rmax}(i)}}{c}}}$

Then, after step 1856 is performed to read entry 1842 in buffer 1840,the receiver controller 1870 can also determine the fire time T(i) forthe subject shot(i) based on the shot timing information 1410 receivedfrom the beam scanner controller 802. Using this shot time as the frameof reference for the TT1 and TT2 offset values, steps 1858 and 1860 canthen operate to start and stop collections from the pixel set at theappropriate times.

FIG. 20 shows an example process flow for use by the signal processingcircuit 1820 to detect returns 1402 and compute return information 1822for the returns 1402. At step 2000, the signal processing circuit 1820digitizes the sensed signal 1806 produced by the photodetector circuitry1800. This sensed signal 1806 represents the incident light on theactivated pixels 1804 of the array 1802 over time, and thus is expectedto include signals corresponding to the returns 1402.

The signal processing circuit 1820 can perform step 2000 using an ADC toproduce digital samples 2004 that are added to buffer 2002.

The signal processing circuit 1820 then needs to segment these samples2004 into groups corresponding to the detection intervals for thereturns from each shot. This aspect of the process flow is identified bythe detection loop 2020 of FIG. 20. A processor 2202 within the signalprocessing circuit 1820 can perform this detection loop 2020. To helpaccomplish this, the processor can access the buffer 1840 to determinethe start and stop collection times for detecting the returns from eachshot. As discussed above, entries 1842 in buffer 1840 can include thestart/stop collection times as either absolute or offset values for TT1and TT2. At step 2006, the processor reads the next entry 1842 in buffer1840 to determine the TT1 and TT2 values (which as noted can be eitherabsolute values or offset values). These TT1 and TT2 values can then beused to find which samples 2004 in the buffer 2002 correspond to thedetection interval of TT1 to TT2 for detecting the subject return from(step 2008). The processor thus reads the digital samples 2004corresponding to the subject detection interval, and then processes thedigital samples 2004 in this group to detect whether the return ispresent (step 2010). When a return is detected, step 2010 can alsocompute return information 1822 based on these samples 2004. As notedabove, the samples 2004 can be processed to compute a range to targetfor the shot. For example, according to a TOF flight technique, theprocessor can compute the range for the return based on knowledge ofwhen the shot was fired, when the detected return was received, and thevalue for the speed of light. The samples 2004 can also be processed tocompute an intensity for the shot return. For example, the returnintensity can be computed by multiplying the return energy by the rangesquared, and then dividing by the transmitted shot energy and again bythe effective receiver pupil aperture. From step 2010, the detectionloop 2020 can return to steps 2006 and 2008 to read the next entry 1842from buffer 1840 and grab the next group of samples 2004 from buffer2002 and then detect the return and compute its return information atstep 2010 (and so on for additional returns).

Multi-Processor Return Detection:

The amount of time needed by processor 2022 to perform the detectionloop 2020 is an important metric that impacts the lidar system. Thisamount of time can be characterized as Tproc, and it defines the rate atwhich processor 2022 draws samples 2004 from buffer 2002. This rate canbe referenced as Rate 1. The rate at which the receiver adds samples2004 to buffer 2002 can be referenced as Rate 2. It is highly desirablefor the processor 2022 to operate in a manner where Rate 1 is greaterthan (or at least no less than) Rate 2 so as to avoid throughputproblems and potential buffer overflows. To improve throughput for thelidar receiver 1400 in this regard, the signal processing circuit 1820can include multiple processors 2022 that distribute the detectionworkload so that the multiple processors 2022 combine to make itpossible for the receiver 1400 to keep up with the shot rate of thelidar transmitter 100 even if Rate 1 is less than the shot rate of thelidar transmitter 100. For example, if there are N processors 2022, thenRate 1 can be N times less than that shot rate of the lidar transmitter100 while still keeping pace with the shots. FIG. 21A shows an exampleof a multi-processor architecture for the signal processing circuit 1820in this regard. As shown by FIG. 21A, the processor 2022 comprises twoor more processors 2022 ₁, . . . , 2022 _(N). Each processor 2022 _(i)can access buffers 1840 and 2002 to perform the operations set forth bysteps 2006-2010 of FIG. 20 for the returns corresponding to differentshots. FIG. 21B shows an example of control flow for the differentprocessors 2022 _(i). At step 2100, each processor 2022 _(i) decideswhether it is free to process another return. In other words, has itfinished processing the previous return it was working on? If thesubject processor 2022 _(i) decides that it is free, it proceeds to step2102 where it performs the detection processing loop 2020 for the nextreturn available from the buffers 1840 and 2002. In this fashion, eachprocessor 2022 _(i) can grab samples 2004 from buffer 2002 to work onthe next return on a first come first served basis and therebydistribute the workload of processing the returns across multipleprocessors to help reduce the processing latency of the signalprocessing circuit 1820.

The processors 2022 _(i) can take any of a number of forms. For example,each processor 2022 _(i) can be a different microprocessor that sharesaccess to the buffers 1840 and 2002. In this fashion the differentmicroprocessors can operate on samples 2004 corresponding to differentreturns if necessary. As another example, each processor 2022 _(i) canbe a different processing core of a multi-core processor, in which casethe different processing cores can operate on samples 2004 correspondingto different returns if necessary. As yet another example, eachprocessor 2022 _(i) can be a different set of parallelized processinglogic within a field programmable gate array (FPGA) orapplication-specific integrated circuit (ASIC). In this fashion,parallelized compute resources within the FPGA or ASIC can operate onsamples 2004 corresponding to different returns if necessary.

It is expected that the use of two processors 2022 will be sufficient todistribute the workload of processing the samples 2004 within buffer2002. With this arrangement, the two processors 2022 can effectivelyalternate in terms of which returns they will process (e.g., Processor 1can work on the samples for even-numbered returns while Processor 2works on the samples for the odd-numbered returns). However, thisalternating pattern may not necessarily hold up if, for example, thedetection interval for Return 1 is relatively long (in which caseProcessor 1 may need to process a large number of samples 2004) whilethe detection intervals for Returns 2 and 3 are relatively short. Inthis example, it may be the case that Processor 1 is still processingthe samples from Return 1 when Processor 2 completes its processing ofthe samples from Return 2 (and thus Processor 2 is free to beginprocessing the samples from Return 3 while Processor 1 is still workingon the samples from Return 1).

Moreover, the return information 1822 computed by each processor 2022_(i) can be effectively joined or shuffled together into their originaltime sequence of shots when adding the return information 1822 to thepoint cloud 1404.

Choosing Rmin, Rmax Values:

The task of choosing suitable Rmin and Rmax values for each shot can betechnically challenging and involves a number of tradeoffs. In an idealworld, the value of Rmin would be zero and the value of Rmax would beinfinite; but this is not feasible for real world applications becausethere are a number of constraints which impact the choice of values forRmin and Rmax. Examples of such constraints are discussed below, andthese constraints introduce a number of tradeoffs that a practitionercan resolve to arrive at desirable Rmin and Rmax values for a given usecase.

For an example embodiment as discussed above where the lidar receiver1400 is only capable of receiving/detecting one return at a time, afirst constraint is the shot timing. That is, the receiver 1400 needs toquit listening for a return from Shot 1 before it can start listeningfor a return from Shot 2. Accordingly, for a given fixed shot spacing,if a practitioner wants to have fixed Rmin and Rmax values, theirdifferences must be equal to the intershot timing (after scaling by2/c). For example, for a 1 μsec detection interval, the correspondingrange buffer would be a total of 150 m. This would permit Rmin to be setat 0 m and Rmax to be set at 150 m (or Rmin=40 m, Rmax=190 m, etc.).Thus, if Rmax is increased, we can avoid adding time to Tproc by alsoincreasing the value of Rmin by a corresponding amount so that theRmax−Rmin does not change.

A second constraint on Rmin, Rmax values is physics. For example, thereceiver 1400 can only detect up to a certain distance for a given shotenergy. For example, if the energy in a laser pulse shot 122 is low,there would not be a need for a large Rmax value. Moreover, the receiver1400 can only see objects up to a certain distance based on theelevation angle. As an example, the receiver 1400 can only see a shortdistance if it is looking at a steep downward elevation angle becausethe field of view would quickly hit the ground at steep downwardelevation angles. In this regard, for a receiver 1400 at a height of 1 mand an elevation angle of −45 degrees, Rmax would be about 1.4 m. Thelight penetration structure of the air within the environment of thelidar system can also affect the physics of detection. For example, ifthe lidar receiver 1400 is operating in clear weather, at night withdark or artificial lighting, and/or in a relatively open area (e.g., ona highway), the potential value for Rmax could be very large (e.g., 1 kmor more) as the lidar receiver 1400 will be capable of detecting targetsat very long range. But, if the lidar receiver 1400 is operating in badweather or during the day (with bright ambient light), the potentialvalue for Rmax may be much shorter (e.g., around 100 m) as the lidarreceiver 1400 would likely only need to be capable of detecting targetsat relatively shorter ranges.

A third constraint arises from geometry and a given use case. Unlike thephysics constraints in the second constraint category discussed above(which are based on features of the air surrounding the lidar system),geometry and use case can be determined a priori (e.g., based on mapsand uses cases such as a given traffic environment that may indicate howcongested the field of view would be with other vehicles, buildings,etc.), with no need to measure attributes in the return data. Forexample, if the goal is to track objects on a road, and the road curves,then there is no need to set Rmax beyond the curve. Thus, if thereceiver 1400 is looking straight ahead and the road curves at a radiusof curvature of 1 km, roughly 100 m for Rmax would suffice. This wouldbe an example where accessing map data can help in the choice ofsuitable Rmax values. As another example, if the lidar receiver 1400 isoperating in a relatively congested environment (e.g., on a citystreet), the potential value for Rmax may be relatively short (e.g.,around 100 m) as the lidar receiver 1400 would likely only need to becapable of detecting targets at relatively short ranges. Also, for usecases where there is some a priori knowledge of what the range is to anobject being targeted with a laser pulse shot, this range knowledge caninfluence the selection of Rmin and Rmax. This would be an example whereaccessing lidar point cloud data 1404 can help in the choice of suitableRmin and Rmax values. Thus, if a given laser pulse shot is targeting anobject having a known estimated range of 50 m, then this knowledge candrive the selection of Rmin, Rmax values for that shot to be values thatencompass the 50 m range within a relatively tight tolerance (e.g.,Rmin=25 m and Rmax =75 m).

A fourth constraint arises from the processing time needed to detect areturn and compute return information (Tproc, as discussed above). Ifthe receiver 1400 has N processors and all are busy processing previousreturns, then the receiver 1400 must wait until one of the processors isfree before processing the next return. This Tproc constraint can makeit undesirable to simply set the detection intervals so that theycoincide with their corresponding shot intervals (e.g., TT1(i)=T(i) andTT2(i)=T(i+1), where TT1(1)=T(1), TT2(1)=T(2), and so on). For example,imagine a scenario where the receiver 1400 includes two processors forload balancing purposes and where the shot spacing has a long delaybetween Shots 1 and 2 (say 100 μsec), and then a quick sequence of Shots2, 3, and 4 (say with intershot spacing of 5 μsec). If Tproc is 2×realtime, then Processor A would need 200 μsec to process the returnfrom Shot 1, and Processor B would need 10 μsec to process the returnfrom Shot 2. This means that Processor A would still be working on Shot1 (and Processor B would still be working on Shot 2) when the returnfrom Shot 3 reaches the receiver 1400. Accordingly, the system may wantto tradeoff the detection interval for detecting the return from Shot 1by using a smaller value for Rmax(1) so that there is a processoravailable to work on the return from Shot 3. Thus, the variable shotintervals that can be accommodated by the lidar system disclosed hereinwill often make it desirable to control at least some of the detectionintervals so that they have durations that are different than thedurations of the corresponding shot intervals, as discussed above.

Accommodating the Tproc constraint can be accomplished in different waysdepending on the needs and desires of a practitioner. For example, undera first approach, the Rmax value for the processor that would be closestto finishing can be redefined to a lesser value so that processor isfree exactly when the new shot is fired. In this case, the Rmin for thenew shot can be set to zero. Under a second approach, we can keep Rmaxthe same for the last shot, and then set Rmin for the new shot to beexactly the time when the processor first frees up. Additional aspectsof this constraint will be discussed in greater detail below.

A fifth constraint arises from the amount of time that the pixels 1804of the array 1802 need to warm up when activated. This can be referredto as a settle time (Tsettle) for the pixels 1804 of the array 1802.When a given pixel 1804 is activated, it will not reliably measureincident light until the settle time passes, which is typically around 1μsec. This settle time effectively defines the average overall firingrate for a lidar system that uses example embodiments of the lidarreceiver 1400 described herein. For example, if the firing rate of thelidar transmitter 100 is 5 million shots per second, the settle timewould prevent the receiver 1400 from detecting returns from all of theseshots because that would exceed the ability of the pixels 1804 to warmupsufficiently quickly for detecting returns from all of those shots.However, if the firing rate is only 100,000 shots per second, then thesettle time would not be a limiting factor.

FIG. 27 shows an example embodiment that illustrates how the circuitryof the receiver 1400 can accommodate the settle time for the pixels1804. For ease of illustration, the array 1802 of FIG. 27 includes onlyfour pixels 1804. However, it should be understood that a practitionerwould likely choose a much larger number of pixels 1804 for the array1802. FIG. 27 shows an example where the photodetector circuit 1800includes an amplifier network that connects the pixels 1804 of array1802 to a corresponding amplifier that amplifies the signals from itscorresponding pixel 1804. Each amplified signal is then fed as an inputline to multiplexer 2710. Thus, as shown by FIG. 27, Pixel 1 feeds intoamplifier A1, which in turn feeds into multiplexer input line M1.Similarly, Pixel 2 feeds into amplifier A2, which in turn feeds intomultiplexer input line M2 (and so on for Pixels 3 and 4). As discussedin the above-referenced and incorporated U.S. Pat. Nos. 9,933,513 and10,754,015, the amplifiers in the amplifier network can be maintained ina quiescent state when their corresponding pixels 1804 are not beingused to detect returns. In doing so, the amount of power consumed by thereceiver 1400 during operation can be greatly reduced. When it is timefor a given pixel 1804 to be used to detect a return, that pixel'scorresponding amplifier is awakened by powering it up. However, asdiscussed above, the pixel will need to wait for the settle time beforeits corresponding amplifier can pass an accurate signal to themultiplexer 2710. This means that if the lidar receiver 1400 is to startcollection from a pixel at time TT1, then it must activate that pixel atleast Tsettle before TT1.

Multiplexer 2710 operates to read out a sensed signal from a desiredpixel 1804 in accordance with a readout control signal 2708, where thereadout control signal 2708 controls which of the multiplexer inputlines are passed as output. Thus, by controlling the readout controlsignal 2708, the receiver 1400 can control which of the pixels 1804 areselected for passing its sensed signal as the return signal 1806.

The receiver controller 1810 includes logic 2700 that operates on thescheduled shot information 1812 to convert the scheduled shotinformation into data for use in controlling the photodetector circuit1800. The scheduled shot information 1812 can include, for each shot,identifications of (1) a shot time (T(i)), (2) shot angles (e.g., anelevation angle, azimuth angle pair), (3) a minimum detection rangevalue (Rmin(i)), and (4) a maximum detection range value (Rmax(i)).Logic 2700 converts this scheduled shot information into the followingvalues used for controlling the photodetector circuit 1800:

-   -   An identification of the pixel set that will be used to detect        the return from the subject Shot (i). This identified pixel set        is shown as P(i) by FIG. 27.    -   An identification of an activation time that will be used to        define the time at which the amplifier(s) corresponding to the        identified pixel set P(i) will be switched to a powered up state        from a quiescent state. This identified activation time is shown        as Ta(i) by FIG. 27.    -   An identification of the start collection time (TT1(i)) for the        identified pixel set P(i)    -   An identification of the stop collection time (TT2(i)) for the        identified pixel set P(i)    -   An identification of a deactivation time that will be used to        define the time at which the amplifier(s) corresponding to the        identified pixel set P(i) will be switched from the powered-up        state to the quiescent state.

The logic 2700 can also pass the shot times T(i) as shown by FIG. 27.

The values for P(i) can be determined from the shot angles in thescheduled shot information 1812 based on a mapping of shot angles topixel sets, as discussed in the above-referenced and incorporatedpatents.

The values for Ta(i) can be determined so that the settle time for theidentified pixel set P(i) will have passed by the time TT1(i) arrives sothat P(i) will be ready to have collection started from it at timeTT1(i). A practitioner has some flexibility in choosing how the logic2700 will compute an appropriate value for Ta(i). For example, the logic2700 can activate the next pixel set when the immediately previous shotis fired. That is, logic 2700 can set the value for Ta(i)=T(i−1), whichis expected to give P(i) enough time to power up so that collection fromit can begin at time TT1(i). However, in another example embodiment, thelogic 2700 can set the value for Ta(i)=TT1(i)−Tsettle (or some timevalue between these two options).

The values for TT1(i) and TT2(ii) can be computed from the Rmin(i) andRmax(i) values as discussed above.

The values for Td(i) can be determined so that Td(i) either equalsTT2(i) or falls after TT2(i), preferably sufficiently close in time toTT2(i) so as to not unduly waste power. In choosing a suitable value forTd(i), the logic 2700 can examine the upcoming shots that are close intime to see if any of the pixels in P(i) will be needed for suchupcoming shots. In such a circumstance, the logic 2700 may choose toleave the corresponding amplifier powered up. But, in an exampleembodiment where a practitioner wants to power down the amplifier(s) fora pixel set as soon as collection from that pixel set stops, then TT2(i)can be used as the deactivation time in place of a separate Td(i) value.

In the example of FIG. 27, the receiver controller 1810 includes pixelactivation control logic 2702 and pixel readout control logic 2704.Pixel activation control logic 2702 operates to provide an activationcontrol signal 2706 to the amplifier network that (1) activates theamplifier(s) corresponding to each pixel set P(i) at each time Ta(i) and(2) deactivates the amplifier(s) corresponding to each pixel set P(i) ateach time Td(i) (or TT2(i) as the case may be). Pixel readout controllogic 2704 operates to provide a readout control signal 2708 to themultiplexer 1710 that operates to (1) select each pixel set P(i) forreadout at time TT1(i) (to begin collection from that pixel set) and (2)de-select each pixel set P(i) at time TT2(i) (to stop collection fromthat pixel set).

Accordingly, FIG. 27 shows an example of how the receiver controller1810 can control which pixel sets of array 1802 will pass their sensedsignals as output in the return signal 1806 to be processed by thesignal processing circuit 1820. To facilitate this timing control, thereceiver controller 1810 can include a pipeline of time slots that arepopulated with flags for the different control signals as may beapplicable, so that the logic 2702 and 2704 can adjust their respectivecontrol signals 2706 and 2708 as may be appropriate as the flags come upas time marches on. By activating pixel sets sufficiently prior to whencollection from them is to begin for return detection (in view ofTsettle), the receiver 1400 is better able to support close range returndetections. For example, if the receiver 1400 were to wait until TT1 toactivate a given pixel set, this would mean that the pixel set would notbe ready to begin detection until the time TT1+Tsettle. Given that atypical value of Tsettle is around 1 μsec, this would translate to aminimum detection range of 150 m. By contrast, using the pixelactivation technique of FIG. 27, the receiver 1400 can support minimumdetection ranges of 0 m.

With an example embodiment, the system begins with the shot list andthen chooses a suitable set of Rmin and Rmax values for each shot. Ofthe five constraints discussed above, all but the second and thirdconstraints discussed above can be resolved based simply on the shotlist, knowledge of Tproc, and knowledge of the number (N) of processors2022 used for load balancing. For example, the third constraint wouldneed access to additional information such as a map to be implemented;while the second constraint would need either probing of the atmosphereor access to weather information to ascertain how air quality mightimpact the physics of light propagation.

In an example embodiment discussed below for computing desired Rmin,Rmax values, the approach balances the first and fourth constraintsusing a mathematical framework, but it should be understood that thisapproach is also viable for balancing the other constraints as well.

FIG. 22 shows an example process flow for assigning Rmin and Rmax valuesto each shot that is scheduled by the control circuit 106. In thisexample, the control circuit 106 (e.g., system controller 800) canperform the FIG. 22 process flow. However, it should be understood thatthis need not necessarily be the case. For example, a practitioner maychoose to implement the FIG. 22 process flow within the receivercontroller 1810 if desired.

The FIG. 22 process flow can operate on the shot list 2200 that isgenerated by the control circuit 106. This shot list 2200 defines aschedule of range points to be targeted with the laser pulse shots 122,where each range point can be defined by a particular angle pair{azimuth angle, elevation angle}. These shots can also be associatedwith a scheduled fire time for each shot.

As discussed above, a number of tradeoffs exist when selecting Rmin andRmax values to use for detecting each shot. This is particularly thecase when determining the detection interval in situations where thereis little a priori knowledge about the target environment. Step 2202 ofFIG. 22 can assign Rmin and Rmax values to each shot based on ananalysis that balances a number of constraints that correspond to thesetradeoffs. This analysis can solve a cost function that optimizes thedetection intervals based on a number of constraints, including thefirst constraint discussed above for an example embodiment where thereceiver 1400 cannot listen to two returns from two different shots atthe same time. Thus it is desirable to optimize the maximum and minimumranges across a given shot list according to a preset cost function,where inputs to the cost function can involve information from the pointcloud data 1404 of previous frames. In general, the cost function caninclude a multiple range function (e.g., a one-to-many or a many-to-manyfunction) resulting in multiple criteria optimization.

The shot list 2200 that step 2202 operates on can be defined in any of anumber of ways. For example, the shot list 2200 can be a fixed list ofshots that is solved as a batch to compute the Rmin, Rmax values. Inanother example, the shot list 2200 can be defined as a shot patternselected from a library of shot patterns. In this regard, the lidarsystem may maintain a library of different shot patterns, and thecontrol circuit 106 can select an appropriate shot pattern based ondefined criteria such as the environment or operational setting of thelidar system. For example, the library may include a desired defaultshot pattern for when a lidar-equipped vehicle is traveling on a highwayat high speed, a desired default shot pattern for when a lidar-equippedvehicle is traveling on a highway at low speed, a desired default shotpattern for when a lidar-equipped vehicle is traveling in an urbanenvironment with significant traffic, etc. Other shot patterns mayinclude foviation patterns where shots are clustered at a higherdensities near an area such as a road horizon and at lower densitieselsewhere. Examples of using such shot pattern libraries are describedin the above-referenced and incorporated U.S. Pat. App. Pub.2020/0025887. Step 2202 can then operate to solve for suitable Rmin,Rmax values for each of the shots in the selected shot pattern.

With respect to step 2202, the plurality of criteria used foroptimization might include, for example, minimizing the range offsetfrom zero meters in front of the lidar receiver 1400, or minimizing therange offset from no less than “x” meters in front of the lidar receiver1400 (where “x” is a selected preset value). The cost function mightalso include minimizing the maximum number of shots in the shot listthat have a range beyond a certain preset range “xx”. In general, “x”and “xx” are adapted from point cloud information in a data adaptivefashion, based on detection of objects which the perception stackdetermines are worthy of further investigation. While the perceptionstack may in some cases operate at much slower time scales, the presetscan be updated on a shot-by-shot basis.

The value of optimization of the range buffers (specifically controllingwhen to start and stop collection of each return) to include multiplerange buffers per scan row is that this allows faster frame rates byminimizing dead time (namely, the time when data is not being collectedfor return detection). The parameters to be optimized, withinconstraints, include processing latency, start time, swath (stop timeminus start time), and row angle offsets. Presets can include statespace for the processor 2022, state space for the laser source 102(dynamic model), and state space for the mirror 110.

Step 2202 solves equations for choosing range buffers (where examples ofthese equations are detailed below), and then generates the range buffer(Rmin and Rmax values) for each shot return. These operations arepre-shot-firing.

The outer bounds for Rmin and Rmax for each shot return can correspondto the pixel switching times TT1 and TT2, where TT1(k) can be set equalto TT2(k−1) and where TT2(k) can be set equal to TT1(k+1). It will oftenbe the case where it is desirable for the lidar receiver 1400 to turnoff the old pixel set at exactly the time the new pixel set is turnedon.

A set of constraints used for a state space model can be described asfollows, for a use case where two processors 2022 are employed toequally distribute the processing workload by handling alternatingreturns.

We assume that the signal processing circuit 1820 begins processing datathe moment the initial data sample is available (namely, at timeTT1(k)). Processor A cannot ingest more data until the processing forReturn(k) is cleared, which we can define as Tproc seconds after theprevious return detection was terminated. The same goes for Processor B.For ease of conception, we will define Tproc as being one half of therealtime rate of return detection (or faster). We will take TT1(k)=T(k)(where an Rmin of zero is the starting point) to simplify thediscussion, although it should be understood that this need not be thecase. With the TT1 values set equal to the fire times of theircorresponding shots, this means that the shot T(k+1) cannot be fireduntil the system stops collecting samples from the last shot. In otherwords, T(k+1)>TT2(k).

Collection for the shot fired at T(k+2) cannot be started unless theprevious shot processed by the same processor (e.g., the same even orodd parity if we assume the two processors 2022 alternate returncollections). This leads to the second of our two inequalities:

$\left. {{T\left( {k + 2} \right)} > {2\left( {{{TT}2(k)} - {T(k)}} \right)}}\rightarrow{\frac{{T\left( {k + 2} \right)} + {T(k)}}{2} > {{TT}2(k)}} \right.,\left. \rightarrow{{{TT}2(k)} \leq {\min\left( {{T\left( {k + 1} \right)},\frac{{T\left( {k + 2} \right)} + {T(k)}}{2}} \right)}} \right.$

If we put together these equations, using ≥≈>, adding relaxationconstraints, and using S as a shift operator (where ST(k)=T(k+1), weget:

${{{{TT}2(k)} + b_{k}} = {{ST}(k)}},{b_{k} \geq 0},{\frac{{S^{2}{T(k)}} + {T(k)}}{2} = {{{TT}2(k)} + b_{k}^{\prime}}},b_{k},{b_{k}^{\prime} \geq 0}$

These inequalities can be re-expressed using matrix notation as shown byFIG. 23A. As noted, S is a shift operator where ST(k)=T(k+1), and S²denotes a shift from T(k) to T(k+2). TT2 and T are expressed as vectorsof dimension B, and b is a positive arbitrary 2N dimensional vector (ofrelaxation terms, built from shuffled versions of b_(k), b_(k)′ of sizetwice that of TT2. I_(n) is an n-by-n identity matrix where the diagonalvalues are all ones and the other values are all zeros; and O_(n) is ann-by-n matrix of all zeros. Inequality constraints can be replaced withequality constraints using new entries as we have done here. B can bereferred to as a relaxation variable. For example, we can replace x>0with x+b=0 (where b>0). Note that T is known, and TT2 is our freevariable.

The equation of FIG. 23A is a state space equation since it is expressedin terms of relations between past and current values on an unknownvalue. This can be solved for real valued variables using simultaneouslinear inequality solvers such as quadratic programming, which isavailable with software packages such as MATLAB (available fromMathworks). An example quadratic programming embodiment for the statespace model of FIG. 23A is shown by FIG. 23B. In this fashion, at step2202 of FIG. 22, the control circuit 106 can determine the detectioninterval data for each of a plurality of laser pulse shots according toa state space equation that is solved using multiple simultaneousinequality constraint equations. This can yield the scheduled shotinformation 1812 where the schedule of range points to be targeted withlaser pulse shots is augmented with associated detection interval datasuch as pixel set activation/deactivation times and/or Rmin, Rmaxvalues. While these discussions are expressed in terms of TT2 values, itshould be understood that these solutions can also work from Rmax valuesusing the relationships discussed above where TT2 can be expressed interms of Rmax and the shot times.

FIG. 23A expresses all possible detection intervals consistent with theshot list, and FIG. 23B represents one solution amongst thesepossibilities (where this solution is one that, loosely speaking,performs uniformly well). But, this solution is not necessarily optimal.For example, at a shot elevation that is low, the ground can be expectedto be close, in which it case it makes little sense to set TT2 in afashion that enables long range detection. A toy example can helpillustrate this point.

Suppose our shot list has shot times in a sequence of 1 μsec, 2 μsec, 98μsec, 100 μsec, 102 μsec.

If we have two processors, each of which computes detections at 2×realtime, we might have as a solution (where we will assume in all casesthat Rmin=0):

Processor A: Shot Time: 1 μsec pulse 98 μsec pulse Range Interval: Rmax= 7.3 km Rmax = 150 m Processor B: Shot Time: 2 μsec pulse 100 μsecpulse Range Interval: Rmax = 7.3 km Rmax = 150 m

While this solution “works”, it should be understood that the two largeRmax values (>7 km) would “hog” the processors by making themunavailable for release to work on another return for awhile. This mightnot be ideal, and one might want to adjust the solution for a smallerRmax value. There are an almost endless set of reasons why this isdesirable because the processors are used for a variety of functionssuch as: intensity computation, range computation, velocity estimation,bounding box estimation etc.

Accordingly, the inventors also disclose an embodiment that combinesmathematical optimization functions with some measure of valuesubstitutions and updating in certain circumstances to arrive a bettersolution (an example of which is discussed below in connection with FIG.24).

As another example where range substitutions and optimization updatescan improve the solution, suppose the shot list obtained for aparticular scenario fires at the following times in units ofmicroseconds, at elevation angle shown respectively:

-   -   Shot Time (μsec)={12 40 70 86 101 121}    -   Elevation-Angle (degrees)={−10,−10,0,0,0,0}

Using the inequality for TT2 above and picking the largest detectioninterval at each shot, the result for the first four shots is:

TT2(1)≤40, TT2(2)≤63, TT2(3)≤85.5, TT2(4)≤101

This maps to detection intervals (in μsec of time) of:

{TT2(k)−T(k)}_(k=1,2,3,4)={28, 23, 15.5, 15}

The sub-optimal nature of this solution arises because it yields largedetection intervals at low elevations (where a long detection intervalis not needed) and a small detection intervals at the horizon (whereelevation angle is zero degrees, which is where a long detectioninterval is more desirable).

As a solution to this issue, FIG. 24 discloses a process flow where thecontrol circuit 106 can swap out potentially suboptimal detectionintervals for more desirable detection intervals. As shown by FIG. 24,step 2202 can operate as described in connection with FIG. 22 togenerate data corresponding to the detection interval solutions computedin accordance with the models of FIGS. 23A and 23B.

The control circuit 106 can also maintain a list 2402 of range pointswith desired detection intervals. For example, the list 2402 canidentify various shot angles that will intersect with the ground withinsome defined distance from the lidar system (e.g., some nominally shortdistance). For a lidar-equipped vehicle, examples of such shot angleswould be for shots where the elevation angle is low and expected to bepointing at the road within some defined distance. For these shotangles, the detection interval corresponding to Rmax need not be a largevalue because it will be known that the shot will hit the ground withinthe defined distance. Accordingly, for these low elevation angles, thelist 2402 can define a desired Rmax or TT2 value that reflects theexpected distance to ground. As another example, the list 2402 canidentify shot angles that lie off the motion path of the lidar system.For example, for a lidar-equipped vehicle, it can be expected thatazimuth angles that are large in absolute value will be looking well offto the side of the vehicle. For such azimuth angles, the system may notbe concerned about potential targets that are far away because they donot represent collision threats. Accordingly, for these large absolutevalue azimuth angles, the list 2402 can define a desired Rmax or TT2value that reflects the shorter range of potential targets that would beof interest. Range segmentations that can be employed by list 2402 mayinclude (1) shot angles linked to desired Rmax or TT2 valuescorresponding to 0-50 m, (2) shot angles linked to desired Rmax or TT2values corresponding to 50-150 m, and (3) shot angles linked to desiredRmax or TT2 values corresponding to 150-300 m.

Then, at step 2404, the control circuit 106 can compare the assigneddetection interval solutions produced by step 2202 with the list 2402.If there are any shots with assigned detection interval solutions thatfall outside the desired detection intervals from list 2402, the controlcircuit 106 can then swap out the assigned detection interval for thedesired detection interval from list 2402 (for each such shot). Thus,step 2404 will replace one or more of the assigned detection intervalsfor one or more shots with the desired detection intervals from the list2402.

The control circuit 106 can then proceed to step 2406 where itre-assigns detection intervals to the shots that were not altered bystep 2404. That is, the shots that did not have their detectionintervals swapped out at step 2404 can have their detection intervalsre-computed using the models of FIGS. 23A and 23B. But, with step 2406,there will be fewer free variables because one or more of the shots willalready have defined detection intervals. By re-solving the state spaceequation with the smaller set of shots, more optical detection intervalsfor those shots can be computed because there will be more spaceavailable to assign to shots that may benefit from longer detectionintervals.

For example, we can re-consider the toy example from above in thecontext of the FIG. 24 process flow. In this example, step 2404 willoperate to impose a 50 m value for Rmax on the shots targeting theelevation of −10 degrees. This 50 m value for Rmax translates to around0.3 μsec. This relaxes the −10 degree cases to:

TT2(1)=T(1)+0.3=12.3

TT2(2)=40.3

This means that both processors 2022 are free when Shot 3 is taken attime 70 (where Shot 3 is the first shot at the horizon elevation, whosedetection interval we wish to make long). The FIG. 24 process flow canmake Shot 3 collect until time 85.5, which frees up a processor 2022(say Processor A) at time 101, just in time to collect on Shot 5. Thenext shot begins at time 86, whereupon Processor B is free, andProcessor B can process 17.5 μsec of data and still free up before theshot at time 121 arrives.

This yields the following for the toy example with respect to FIG. 24:

-   -   Shot Time (μsec)={12 40 70 86 101 121}    -   Elevation-Angle (degrees)={−10,−10,0,0,0,0}    -   Detection Intervals: {TT2(k)−T(k)}_(k=1,2,3,4)={0.3, 0.3, 15.5,        17.5}

We can see that the FIG. 24 process flow has increased the detectioninterval for the zero degree elevation shots at the expense of those at−10 degrees in elevation, which provides a better set of detectionintervals for the shot list.

Lidar System Deployment:

The inventors further note that, in an example embodiment, the lidarreceiver 1400 and the lidar transmitter 100 are deployed in the lidarsystem in a bistatic architecture. With the bistatic architecture, thereis a spatial offset of the field of view for the lidar transmitter 100relative to the field of view for the lidar receiver 1400. This spatialseparation provides effective immunity from flashes and first surfacereflections that arise when a laser pulse shot is fired. For example, anactivated pixel cluster of the array 1802 can be used to detect returnsat the same time that the lidar transmitter 100 fires a laser pulse shot122 because the spatial separation prevents the flash from the newlyfired laser pulse shot 122 from blinding the activated pixel cluster.Similarly, the spatial separation also prevents the receiver 1400 frombeing blinded by reflections from surfaces extremely close to the lidarsystem such as glass or other transparent material that might be locatedat or extremely close to the egress point for the fired laser pulse shot122. An additional benefit that arises from this immunity to shotflashes and nearby first surface reflections is that it permits thebistatic lidar system to be positioned in advantageous locations. Forexample, in an automotive or other vehicle use case as shown by FIG. 25,the bistatic lidar system 2500 can be deployed inside aclimate-controlled compartment 2502 of the vehicle 2504 (such as thepassenger compartment), which reduces operational risks to the lidarsystem arising from extreme temperatures. For example, theair-conditioning inside the compartment 2502 can reduce the risk of thelidar system 2500 being exposed to excessive temperatures. Accordingly,FIG. 25 shows an example where the bistatic lidar system 2500 isdeployed as part of or connected to a rear view mirror assembly 2510 orsimilar location in compartment 2502 where the lidar system 2500 canfire laser pulse shots 122 and detect returns 1402 through the vehicle'swindshield 2512. It should be understood that the components of FIG. 27are not shown to scale.

Multi-Channel Readout for Returns:

For another example embodiment, it should be understood that thedetection timing constraint discussed above where the detectionintervals are non-overlapping can be removed if a practitioner choosesto deploy multiple readout channels as part of the photodetectorcircuitry 1800, where these multiple readout channels are capable ofseparately reading the signals sensed by different activated pixelclusters at the same time. FIG. 26 shows an example receiver 1400 thatincludes multiple readout channels (e.g., M readout channels). Eachreadout channel can include a multiplexer 2600 that reads the signalssensed by a given activated cluster of pixels 1804, in which case thelidar receiver 1400 is capable of detecting returns that impactdifferent pixel clusters of the array 1802 at the same time. It shouldbe understood that amplifier circuitry can be placed between the array1802 and the multiplexers 2600 as described above with reference to FIG.27 and as described in the above-referenced and incorporated U.S. Pat.Nos. 9,933,513 and 10,754,015. Through the use of multiple readoutchannels as exemplified by FIG. 26, practitioners can relax theconstraint that the detection intervals for detecting returns fromdifferent shots be non-overlapping. Among other benefits, this approachcan open up possibilities for longer range detections that mightotherwise be missed because collections from a first pixel clusterneeded to detect the long range return would have stopped so thereceiver 1400 could start collection from a second pixel cluster neededto detect a shorter range return. With the approach of FIG. 26,collections from the first pixel cluster can continue for a longer timeperiod, even when collections are occurring from the second pixelcluster through a different readout channel, thereby enabling thedetection of the longer range return.

While the invention has been described above in relation to its exampleembodiments, various modifications may be made thereto that still fallwithin the invention's scope.

For example, while the example embodiments discussed above involve amirror subsystem architecture where the resonant mirror (mirror 110) isoptically upstream from the point-to-point step mirror (mirror 112), itshould be understood that a practitioner may choose to position theresonant mirror optically downstream from the point-to-point stepmirror.

As another example, while the example mirror subsystem 104 discussedabove employs mirrors 110 and 112 that scan along orthogonal axes, otherarchitectures for the mirror subsystem 104 may be used. As an example,mirrors 110 and 112 can scan along the same axis, which can then producean expanded angular range for the mirror subsystem 104 along that axisand/or expand the angular rate of change for the mirror subsystem 104along that axis. As yet another example, the mirror subsystem 104 caninclude only a single mirror (mirror 110) that scans along a first axis.If there is a need for the lidar transmitter 100 to also scan along asecond axis, the lidar transmitter 100 could be mechanically adjusted tochange its orientation (e.g., mechanically adjusting the lidartransmitter 100 as a whole to point at a new elevation while mirror 110within the lidar transmitter 100 is scanning across azimuths).

As yet another example, a practitioner may find it desirable to drivemirror 110 with a time-varying signal other than a sinusoidal controlsignal. In such a circumstance, the practitioner can adjust the mirrormotion model 308 to reflect the time-varying motion of mirror 110.

As still another example, it should be understood that the techniquesdescribed herein can be used in non-automotive applications. Forexample, a lidar system in accordance with any of the techniquesdescribed herein can be used in vehicles such as airborne vehicles,whether manned or unmanned (e.g., airplanes, drones, etc.). Furtherstill, a lidar system in accordance with any of the techniques describedherein need not be deployed in a vehicle and can be used in any lidarapplication where there is a need or desire for hyper temporal controlof laser pulses and associated lidar processing.

These and other modifications to the invention will be recognizable uponreview of the teachings herein.

What is claimed is:
 1. A lidar system comprising: a photodetector array,the photodetector array comprising a plurality of pixels for sensingsignals representative of incident light on the pixels; a plurality ofreadout channels from the photodetector array, wherein the readoutchannels permit a plurality of readouts of sensed signals from differentsets of the pixels at the same time; a signal processing circuit; and acontrol circuit; wherein the control circuit determines a plurality ofdetection intervals for controlling the signal processing circuit todetect returns from a plurality of laser pulse shots via different setsof the pixels; and wherein the signal processing circuit processes aplurality of sensed signals read out from the photodetector array to (1)detect the returns and (2) compute return information based on thedetected returns; and
 2. The system of claim 1 wherein the readoutchannels allow for control circuit to define a plurality of detectionintervals that overlap.
 3. The system of claim 1 wherein a plurality ofthe detection intervals are overlapping.
 4. The system of claim 1wherein the control circuit identifies the different pixel sets to readout from based on a plurality of range points that are targeted by thelaser pulse shots.
 5. The system of claim 4 wherein the shot listidentifies the targeted range points for the laser pulse shots byazimuth and elevation angles.
 6. The system of claim 1 wherein each ofthe different pixel sets comprises one or more of the pixels of thearray.
 7. The system of claim 1 wherein each determined detectioninterval has a corresponding pixel set from which sensed signals areread out, and wherein each of a plurality of the determined detectionintervals comprises (1) first data that indicates when to startcollection from its corresponding pixel set and (2) second data thatindicates when to stop collection its corresponding pixel set.
 8. Thesystem of claim 7 wherein, for each of a plurality of the determineddetection intervals, the first and second data comprise estimates ofminimum and maximum ranges for the range point targeted by a laser pulseshot associated with that determined detection interval.
 9. The systemof claim 8 wherein the control circuit translates the minimum andmaximum range estimates into start and stop collection times for thepixel sets corresponding to the determined detection intervals.
 10. Thesystem of claim 1 wherein the control circuit activates pixels of thearray to be used for detecting the returns sufficiently prior to whencollections are to start from the activated pixels for a pixel settletime to have passed when the collections are to start from the activatedpixels.
 11. The system of claim 10 wherein the signal processing circuitcomprises a plurality of processors that share processing of the sensedsignals.
 12. The system of claim 10 wherein the signal processingcircuit updates a lidar point cloud with the computed returninformation.
 13. The system claim 1 further comprising: a lidartransmitter, wherein the lidar transmitter comprises a scannable mirror,and wherein the lidar transmitter transmits the laser pulse shots towardtargeted range points via the scannable mirror.
 14. The system of claim13 wherein the lidar transmitter scans the scannable mirror in aresonant mode.
 15. The system of claim 14 wherein the lidar transmitterscans the scannable mirror in the resonant mode at a scan frequency in arange between 100 Hz and 20 kHz.
 16. The system of claim 14 wherein thelidar transmitter scans the scannable mirror in the resonant mode at ascan frequency in a range between 10 kHz and 15 kHz.
 17. The system ofclaim 13 wherein the scannable mirror comprises a first scannable mirrorand a second scannable mirror, wherein the lidar transmitter transmitsthe laser pulse shots toward the targeted range points via the first andsecond scannable mirrors.
 18. The system of claim 17 wherein the lidartransmitter scans the second scannable mirror in a point-to-point modeaccording to a step function that varies as a function of the rangepoints targeted with the laser pulse shots.
 19. The system of claim 17wherein the second scannable mirror is optically downstream from thefirst scannable mirror.
 20. The system of claim 13 wherein the lidartransmitter and the photodetector circuit are in a bistatic arrangementwith respect to each other.
 21. The system of claim 13 furthercomprising a laser source that generates the laser pulse shots, andwherein the control circuit schedules the laser pulse shots in the shotlist according to a laser energy model for the laser source.
 22. Thesystem of claim 21 wherein the control circuit schedules the laser pulseshots in the shot list according to the laser energy model and a mirrormotion model for the scannable mirror.
 23. The system of claim 1 whereinthe array comprises a two-dimensional (2D) array of pixels.
 24. A methodcomprising: sensing incident light corresponding to a plurality ofreturns from a plurality of laser pulse shots, wherein the sensing stepis performed by a photodetector array comprising a plurality of pixels;and simultaneously reading sensed signals from a plurality of differentsets of pixels of the photodetector array via a plurality of readoutchannels; and detecting different returns from the laser pulse shotsbased on different sensed signals read out by different readoutchannels.
 25. The method of claim 24 further comprising: determining aplurality of detection intervals that govern when the reading step isperformed by a readout channel to obtain signal data corresponding todifferent returns from different laser pulse shots.
 26. The method ofclaim 25 wherein the determining step comprises determining thedetection intervals based on defined criteria, wherein the definedcriteria includes estimates of ranges for objects to be detected by thelaser pulse shots.
 27. The method of claim 25 wherein the determineddetection intervals include overlapping detection intervals.